search

codemanki / cloudscraper

--DEPRECATED -- 🛑 🛑 Node.js library to bypass cloudflare's anti-ddos page
MIT License
603 stars 141 forks source link

(tls): make ciphers similar to chrome #238

Closed ghost closed 5 years ago

ghost commented 5 years ago

As titled, this change makes the cipher list very similar to what chromium sends.

After comparing the cipher list with the change, these were the only differences according to https://www.howsmyssl.com/a/check

Besides TLS_RSA_WITH_AES <- some differences there, see the lists. (We already omit some SHA-1 via !ECDHE+SHA:!AES128-SHA)

Node sends:

Chrome sends:

The rest was the same on my system:

Notes:

Ciphers omitted due to change ```js [ 'TLS_DHE_RSA_WITH_AES_128_GCM_SHA256', 'TLS_DHE_RSA_WITH_AES_128_CBC_SHA256', 'TLS_DHE_RSA_WITH_AES_256_CBC_SHA256', 'TLS_DHE_DSS_WITH_AES_256_GCM_SHA384', 'TLS_DHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_DHE_RSA_WITH_CHACHA20_POLY1305', 'TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8', 'TLS_ECDHE_ECDSA_WITH_AES_256_CCM', 'TLS_DHE_RSA_WITH_AES_256_CCM_8', 'TLS_DHE_RSA_WITH_AES_256_CCM', 'TLS_ECDHE_ECDSA_WITH_ARIA_256_GCM_SHA384', 'TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384', 'TLS_DHE_DSS_WITH_ARIA_256_GCM_SHA384', 'TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384', 'TLS_DHE_DSS_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8', 'TLS_ECDHE_ECDSA_WITH_AES_128_CCM', 'TLS_DHE_RSA_WITH_AES_128_CCM_8', 'TLS_DHE_RSA_WITH_AES_128_CCM', 'TLS_ECDHE_ECDSA_WITH_ARIA_128_GCM_SHA256', 'TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256', 'TLS_DHE_DSS_WITH_ARIA_128_GCM_SHA256', 'TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256', 'TLS_DHE_DSS_WITH_AES_256_CBC_SHA256', 'TLS_DHE_DSS_WITH_AES_128_CBC_SHA256', 'TLS_DHE_RSA_WITH_AES_256_CBC_SHA', 'TLS_DHE_DSS_WITH_AES_256_CBC_SHA', 'TLS_DHE_RSA_WITH_AES_128_CBC_SHA', 'TLS_DHE_DSS_WITH_AES_128_CBC_SHA', 'TLS_RSA_WITH_AES_256_CCM_8', 'TLS_RSA_WITH_AES_256_CCM', 'TLS_RSA_WITH_ARIA_256_GCM_SHA384', 'TLS_RSA_WITH_AES_128_CCM_8', 'TLS_RSA_WITH_AES_128_CCM', 'TLS_RSA_WITH_ARIA_128_GCM_SHA256' ] ```
Chrome v75 ciphers ```js [ 'TLS_GREASE_IS_THE_WORD_0A', 'TLS_AES_128_GCM_SHA256', 'TLS_AES_256_GCM_SHA384', 'TLS_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA', 'TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA', 'TLS_RSA_WITH_AES_128_GCM_SHA256', 'TLS_RSA_WITH_AES_256_GCM_SHA384', 'TLS_RSA_WITH_AES_128_CBC_SHA', 'TLS_RSA_WITH_AES_256_CBC_SHA', 'TLS_RSA_WITH_3DES_EDE_CBC_SHA' ] ```
Node.js v12.7.0 (This PR) ```js [ 'TLS_AES_256_GCM_SHA384', 'TLS_CHACHA20_POLY1305_SHA256', 'TLS_AES_128_GCM_SHA256', 'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256', 'TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384', 'TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384', 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256', 'TLS_RSA_WITH_AES_256_GCM_SHA384', 'TLS_RSA_WITH_AES_128_GCM_SHA256', 'TLS_RSA_WITH_AES_256_CBC_SHA256', 'TLS_RSA_WITH_AES_128_CBC_SHA256', 'TLS_RSA_WITH_AES_256_CBC_SHA', 'TLS_EMPTY_RENEGOTIATION_INFO_SCSV' ] ```
ghost commented 5 years ago

@codemanki

Perhaps we add this under a flag or two, sslretry or something? I'm not really sure how necessary it is or if it even mitigates the problem at all... As I suspect it largely depends on the system, CPU, etc..

I'm okay with closing this PR but if it can help...IDK :confused:

codemanki commented 5 years ago

I'm not really sure how necessary it is or if it even mitigates the problem at all... As I suspect it largely depends on the system, CPU, etc..

Will this change somehow negatively affect users? If not, then I'm happy to merge it :)

ghost commented 5 years ago

That's been my concern as well and I've chosen to look at it like this, with these settings if cloudscraper can't connect then neither can google chrome. I think it'll help as is but I'm willing to develop this PR more to be on the safe side.

ghost commented 5 years ago
:thinking: Alternatively, we could add a new flag: ```js const cloudscraper = require('cloudscraper').defaults({ tls: 'chrome' }) ``` Where the default would be `node` but users could opt-in to using more chrome-like TLS. It should be noted that the cipher suite of this PR is more secure than the current default-ish one based on node. Or we could check `defaultParams.agentOptions.ciphers` for the value of `chrome`. Or Maybe a mode or browser flag that could be one of `default`, `node`, `chrome`, etc. * default - The current default settings * node - Don't use custom agentOptions, etc * chrome - Make things more chrome-like I think chrome would be the recommended setting so I question whether a flag makes the most sense. This can already be done by the user so it's just sugar. Yet another flag would be added to the already long list of options.
codemanki commented 5 years ago

@pro-src why not to just document this, and let people provide values they need via defaultParams.agentOptions.ciphers ? This looks like very advanced usage to me, so maybe having it in documentation would be sufficient. I do agree that we are already having too many flags :)

ghost commented 5 years ago

why not to just document this, and let people provide values they need via defaultParams.agentOptions.ciphers ?

I'd like to decrease the number of bug reports. Documentation in issues doesn't seem to be understood or users don't look there. So really I'm targeting those that don't understand the problem and/or shouldn't set their own TLS settings. Unfortunately, I think that's the majority of people.

I reiterate that I don't know how many users are receiving CAPTCHA because of Cloudflare's advanced TLS checks or what percentage of those users this will help.

Please see what a chrome-dev had to say about preferring such an option:

Thanks for your feedback. We intentionally do not expose a number of options, particularly those that can cause harm to users through confusion or misunderstanding. Our principles include "simplicity" - which is why we don't expose a number of options, despite knowing that some users may wish to configure them - and "security" - where we want to ensure we're secure out of the box for users.

Configuring cipher suites is in conflict with both of those principles. A whitelist approach is especially dangerous, as it actually harms users by preventing the deployment of new, stronger ciphersuites. I can assure you, Chrome has no plans to implement weaker cipher suites than what it ships now, just as the TLS WG has made it clear they're unlikely to standardize those weaker than what they have now.

I realize there may be some disagreement about our security posture; despite leading the industry in promoting good and strong cryptography, there will always be some who feel we don't do enough. On this topic, if it is of critical importance to you, the best answer I can give is that it may be that Chrome isn't the right fit for your needs

I agree with that to some extent.

The question of whether or not we should do this for the user boils down to a few questions:

I think the answer to those questions is no and that these settings are the recommended.

OTH, if only a few people are affected and this PR may not help them then it's a waste of time.

So if we close it, meh If we merge it, meh

IDK, and I'm leaving up to you.

ghost commented 5 years ago

Tbc, We can definitely close it and just document the setting for now. I'm not against either option because there are some unknown factors. Aka this is really complicated. So we put it on ice and just chill or ???

ghost commented 5 years ago

The reason why I'm not trying to match chrome exactly is that I don't think the remaining differences matter. It's also impossible to match exactly due to the differences between openssl and boringssl. AFAIK, only boringssl supports grease and we don't have a way to tell openssl to stop sending the TLS_EMPTY_RENEGOTIATION_INFO_SCSV which should be sent unless you disable renegotiation.

OpenSSL sucks and Node.js doesn't improve it at all.

codemanki commented 5 years ago

Thank you for such detailed explanation and answer. I think then based on those three no and general uncertainty of how this will affect and if it is gonna improve anything, I would rather close this PR for now.