yifeikong / curl_cffi

Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
https://curl-cffi.readthedocs.io/
MIT License
1.63k stars 210 forks source link

Add support for Firefox #59

Open JodanJodan opened 1 year ago

JodanJodan commented 1 year ago

The latest curl_impersonate supports 6 Firefox versions, could these be added?

codeman5000 commented 9 months ago

Hi, this would also interest me. is there any way we can modify code to do this ourself?

yifeikong commented 9 months ago

Take a look at the makefile, it's fairly simple. Basically, it's just downloading the compiled .so files and then using cibuildwheel to build binary wheels.

codeman5000 commented 9 months ago

Hello,

thx for your email. I'll take a look at it. Your project helps me a lot in bypassing bot protection when doing requests.

greetz

codeman5000 commented 9 months ago

Hello,

i think it is to complicated for me to get it working. i have no idea where these come from:

.preprocessed: curl_cffi/include/curl/curl.h curl_cffi/cacert.pem .so_downloaded touch .preprocessed

any idea if there will be more browser imitations available in the future?

kind regards

gunir commented 9 months ago

+1

I also think Firefox's TLS Fingerprint should also be an option, it's useful to impersonate Firefox in case we use curl_cffi as proxy to surf the web with Firefox too, can't just impersonate Chrome because it's easy to detect.

lxntagram commented 7 months ago

Hello, I'm trying to add firefox. How can I get browser infos like ciphers, and curves?

yifeikong commented 7 months ago

Actually you don’t need to worry about that, it’s all there in the upstream curl-impersonate.

The real problem is that Firefox uses NSS, while other browsers use boringssl, and curl can only be linked to one TLS lib at the same time. So there are two separate libcurl so files for each browsers.

For curl-cffi, it’s also difficult to link to these two different so files. What you need to do is to find a way to add the firefox in addition to the chrome one, while allowing they can be selected by the user at runtime, or at least before importing curl-cffi.

来自手机回复

dIscOrd @.***>于2023年11月14日 周二22:56写道:

Hello, I'm trying to add firefox. How can I get browser infos like ciphers, and curves?

— Reply to this email directly, view it on GitHub https://github.com/yifeikong/curl_cffi/issues/59#issuecomment-1810384367, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH4ZXZPBTVUPMS5BND2SZ3YEOBCNAVCNFSM6AAAAAAYAY6Y7SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJQGM4DIMZWG4 . You are receiving this because you were assigned.Message ID: @.***>

bjia56 commented 6 months ago

Would it be easier to have a sibling package, e.g. curl_cffi_ff, which provides the Firefox version of curl-impersonate? I would think that avoids the duplicate symbols issue

yifeikong commented 6 months ago

Yes, it’s one possible way, I have tried it actually, but it’s not ideal, neither.

I was thinking that is it possible to use boringssl to emulate NSS? Then it would be perfect to add firefox.

来自手机回复

Brett Jia @.***>于2023年12月17日 周日11:11写道:

Would it be easier to have a sibling package, e.g. curl_cffi_ff, which provides the Firefox version of curl-impersonate? I would think that avoids the duplicate symbols issue

— Reply to this email directly, view it on GitHub https://github.com/yifeikong/curl_cffi/issues/59#issuecomment-1859023933, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH4ZX7AKWOZ5SASOWJN6P3YJZPGRAVCNFSM6AAAAAAYAY6Y7SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJZGAZDGOJTGM . You are receiving this because you were assigned.Message ID: @.***>

gunir commented 6 months ago

I was thinking that is it possible to use boringssl to emulate NSS? Then it

I think it's possible, as BoringSSL is very very close to NSS, unlike OpenSSL it's impossible because OpenSSL doesn't allow to hide some ciphers and a lot of extensions.

bjia56 commented 6 months ago

It turns out that ffi is happy build multiple .so wrappers within the curl_cffi package, so it's possible to build curl_cffi._wrapper_chrome and curl_cffi._wrapper_ff at the same time. image Much of the library will need to be tweaked slightly to generalize the assumption that there's only one wrapper. Additionally, Firefox does not support the certificate store: image

yifeikong commented 6 months ago

Cool, do you mind submitting a PR?

bjia56 commented 6 months ago

Yeah I'll probably need a bit of time to clean things up but I'll open a PR soon