search

lexiforest / curl_cffi

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
https://curl-cffi.readthedocs.io/
MIT License
2.4k stars 258 forks source link

[Feature] Android extensions randomize #393

Open Cataphract89 opened 1 month ago

Cataphract89 commented 1 month ago

Is there any way to randomize extensions for Ja3 real android similar to chrome...?

   771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,(...),29-23-24,0

Real androids always have them different, but any manual change of extension order always results in errors

lexiforest commented 1 month ago

Please describe in details about the error. I just tested with 2 random ja3 from chrome, and it works.

>>> ja3 = "771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,13-17513-16-11-0-65037-35-27-10-18-45-5-65281-51-23-43,25497-29-23-24,0"
>>> requests.get("https://tls.browserleaks.com/json", ja3=ja3).json()["ja3_text"] == ja3
True
>>> ja3 = "771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,35-27-23-16-18-65037-11-17513-51-45-10-65281-5-0-13-43,25497-29-23-24,0"
>>> requests.get("https://tls.browserleaks.com/json", ja3=ja3).json()["ja3_text"] == ja3
True
nakoo commented 1 month ago

It sounds like more JA3 issue due to chromium's randomization. https://github.com/salesforce/ja3/issues/88

lexiforest commented 1 month ago

It sounds like more JA3 issue due to chromium's randomization. salesforce/ja3#88

No, Chrome only permutes extensions, which has been known and implemented since the beginning of curl-impersonate and curl_cffi.

Cataphract89 commented 1 month ago

It is about android. I noticed that all the difference between ja3 chrome and android devices is in supported_groups Сhrome128:

     supported_groups = "25497-29-23-24"

Real android

      supported_groups="29-23-24"

When I try to connect to random extensions from ja3 android device, I get cloudflare error. When running chrome, there are no problems

       Failed to perform, curl: (22) Refused WebSockets upgrade: 403

Similarly, there are no problems either way when dealing with requests. It is about logic and possible randomization of android extensions work with socket

lexiforest commented 1 month ago

Random here does not mean "completely random order", there are some rules you have to follow. For example, the supported groups are stored in one extension, if you remove that extension in the TLS hello package, it will not make sense.

Please at least provide one JA3 string that you can not apply, so I can know exactly what your issue is.

Cataphract89 commented 1 month ago

Apparently the problems associated with android extensions are purely with the internal server logic. Now errors occur even with extensions taken from real android devices.

Cataphract89 commented 1 month ago

There is a question related to this topic - when connecting to the server in multithreading, over time errors 403 and 429 start to occur. I use randomly different impersonate, but it doesn't help, the errors go away only if you stop connecting to the socket for some time when taking into account the use of proxies.

Question - are there any ways to randomize impersonate besides user agent, ja3, extra_fp, and akamai for socket?

jorgeupx commented 1 month ago

Hey Cataphract89, any way to reach out to you? interested to see how you handled impersonating with websocket, trying to mock an IOS FP with websocket since apparently there is not a single library no matter what lang that allows this, but you seem to be onto something!

lexiforest commented 1 month ago

It is about android. I noticed that all the difference between ja3 chrome and android devices is in supported_groups Сhrome128:

     supported_groups = "25497-29-23-24"

Real android

      supported_groups="29-23-24"

When I try to connect to random extensions from ja3 android device, I get cloudflare error. When running chrome, there are no problems

       Failed to perform, curl: (22) Refused WebSockets upgrade: 403

Similarly, there are no problems either way when dealing with requests. It is about logic and possible randomization of android extensions work with socket

I cannot reproduce this. Notice the supported group part is set to "29-23-24".

>>> ja3 = "771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,35-27-23-16-18-65037-11-17513-51-45-10-65281-5-0-13-43,29-23-24,0"
>>> requests.get("https://tls.browserleaks.com/json", ja3=ja3).json()["ja3_text"] == ja3
True

Question - are there any ways to randomize impersonate besides user agent, ja3, extra_fp, and akamai for socket?

There are many parts that we cannot change, because it's not supported by libcurl. For example, we cannot impersonate the PSK extension with libcurl, which are used by most browsers.

Besides, servers use other techniques, such as the JS chanllege to identify malform traffic, which is not and cannot be covered by curl_cffi.