lwthiker / curl-impersonate

curl-impersonate: A special build of curl that can impersonate Chrome & Firefox
MIT License
3.82k stars 254 forks source link

Is a correct ja3 hash an indicator of success? #134

Open windbridges opened 1 year ago

windbridges commented 1 year ago

I have to use curl-impersonate in php not directly, but through a symfony CurlHttpClient object. I have followed all the instructions in the manual and everything seems to work correctly. At least when I make a request to https://tls.peet.ws/api/all through the chrome browser and through CurlHttpClient, I get identical results and the same ja3 hash. Is this sufficient proof that curl-impersonate is being used correctly?

And a little off-topic. Despite the fact that everything seems to be configured correctly, one famous site still understands that all my requests are made by the same software. A couple of minutes it allows me to make requests, after that all requests start getting HTTP 429 responses. Each request is done with a new proxy, it is not repeated. If I stop the requests and take a break for 10 minutes, the site allows me to make requests again for a while.

Maybe someone here can tell me by what criteria other than TLS fingerprint, the site can identify the client?

jlcd commented 1 year ago

Regarding the JA3 Hash, I guess it's correct, if it shows the same for the browser and impersonate, you're good to go. Another API worth checking is https://tls.peet.ws/api/all (take a look at the akamai fingerprint).

About the 429, there are many ways a site/WAF generates a fingerprint (canvas/TLS/audio to name a few), and also sometimes they have websockets running, which can bypass the proxy configuration on the browser, so worth taking a look. But in general, there are many ways to fingerprint a website access, just having random IPs and a good TLS FP is not always enough.

windbridges commented 1 year ago

Regarding the JA3 Hash, I guess it's correct, if it shows the same for the browser and impersonate, you're good to go. Another API worth checking is https://tls.peet.ws/api/all (take a look at the akamai fingerprint).

Yes, the akamai fingerprint also matches. I'm just not quite sure about the options --http2 --false-start --compressed --tlsv1.2 --no-npn --alps --cert-compression brotli that are set from curl_chrome* files, whether they apply. I set env CURL_IMPERSONATE=chrome101 when I run the script. But I'm not sure if this file is applied when curl is initialized through the php function curl_init in CurlHttpClient. Will ja3 match if any of these options did not apply?

About the 429, there are many ways a site/WAF generates a fingerprint (canvas/TLS/audio to name a few), and also sometimes they have websockets running, which can bypass the proxy configuration on the browser, so worth taking a look. But in general, there are many ways to fingerprint a website access, just having random IPs and a good TLS FP is not always enough.

Fingerprinting canvas/audio/fonts as well as websockets, WebRTC and so on, these are technologies that only become available if we make a request through a browser. And if we make a request through curl from a new ip address, what can the site use other than IP, user agent, HTTP header order, and TLS?

vhpm18 commented 1 year ago

CurlHttpClient Hey bro, could you help me, I can't find how to make php load curl-impersonate, I got it to work, here it is

Array ( [version_number] => 480256 [age] => 9 [features] => 1370063517 [ssl_version_number] => 0 [version] => 7.84.0 [host] => x86_64-pc-linux-gnu [ssl_version] => BoringSSL [libz_version] => 1.2.11 [protocols] => Array ( [0] => dict [1] => file [2] => ftp [3] => ftps [4] => gopher [5] => gophers [6] => http [7] => https [8] => imap [9] => imaps [10] => mqtt [11] => pop3 [12] => pop3s [13] => rtsp [14] => smb [15] => smbs [16] => smtp [17] => smtps [18] => telnet [19] => tftp )

[ares] => 
[ares_num] => 0
[libidn] => 
[iconv_ver_num] => 0
[libssh_version] => 
[brotli_ver_num] => 16777225
[brotli_version] => 1.0.9

)

but I can't get php to load it to make the requests when I go to make the request it takes curl by default that I have installed, I'm using it from Laravel