Closed Anorov closed 5 years ago
@lukele I know you've done a lot of testing already but do you mind trying with and without the TLSv1.3 ciphers. If there isn't any difference then I think it's safe to drop them unless proven otherwise.
Was close to losing it here, since I've remembered that I did have a version which worked with python2 and openssl <= 1.1.1
:!ECDHE+SHA
doesn't work
:!SHA1
doesn't work
:!AES128-SHA
does work
So the following cipher suite modification works for me with Python 2.7.15/OpenSSL 1.0.2r
and Python 3.7.3/OpenSSL 1.1.1b
:!ECDHE+SHA:!AES128-SHA
@pro-src This is the list without TLSv1.3, correct?
ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5:!ECDHE+SHA
Yup, I think so.
That ends with a captcha.
Adding !AES128-SHA
fixes it again
Since it appears that OpenSSL 1.1.0 did have some kind of partial TLS1.3 support
Wrong. There is absolutely no TLSv1.3 support, not even partial, in OpenSSL 1.1.0.
Yes, I believe to have misunderstood this. Only reference I could find was: https://securityinaction.wordpress.com/2016/09/13/openssl-1-1-0-adds-partial-tls-1-3-support/
Regardless, in my limited testing, python versions linked against OpenSSL < 1.1.1 seem to work now as well, by removing AES128-SHA
@lukastribus
I don't know anything about nodejs, but it seems to me that you just configured a large cipher list that happened to contain TLSv1.3 ciphers, and because it works, you assumed the TLSv1.3 ciphers did the job. But there is more in that configuration than TLSv1.3 ciphers and you did not make the opposite test: the same list without TLSv1.3 ciphers.
I understand you guys have this impression, it's wrong. Here's my last attempt at convincing you:
Why? Did you not read this comment? https://github.com/Anorov/cloudflare-scrape/issues/235#issuecomment-492035224
I have the impression that you're stuck arguing about those ciphers. I don't believe that you've actually helped solve the current problem at all. Please focus on the issue at hand.
@lukele The PR looks great and is a minimal solution. :tada:
I'm still curious about that SSLv3 handshake error. The adapter can be toggled on and off so I'm not really worried about it, just curious.
Are you still seeing that error?
I just tested against ssllabs on python 2 and 3: <Response [200]>
(No errors)
@lukele I updated the script. If you want to generate a full report so we'll have something to reference, all of the hashes were removed.
Note that by normal conditions, it actually means your PR.
Here's mine for reference: https://gist.github.com/pro-src/c1c464e394e4d25b81b16a1991b72316
@pro-src should I run the script with python3 and python2?
Maybe. I want all the information that we can get but I imagine that the results will be the same so long as the same package versions are used. If you add the hashes from the first run, you'll save us from having to compare the results of the second.
Unfortunately python2 doesn't support SSLSocket.shared_ciphers() and I couldn't quickly find a way to get to that information.
This is my report from python3: https://gist.github.com/lukele/8d00ad380bb70e27dc43f8e7f3d57472
Hashes for DEFAULT ciphers hash
, Signature algorithms hash
don't match.
Also, why does your report have
https://pro-src.com TLSv1.1 ciphers hash: 51f39199142f9474451bf94a47f70a46
but my report has
https://pro-src.com DEFAULT :!ECDHE+SHA:!AES128-SHA ciphers hash: 59fac59572e9abb2cdf770e0debfdfa3
Thanks for generating the report.
There is only one way to check the ciphers when sending requests to https://pro-src.com whereas there is two ways when requests are made to https://howmyssl.com and https://www.ssllab.coms. The hashes are appended to the list of known hashes as soon they're encountered to avoid duplicates even when there isn't any known initially. There is more requests listed in the report than cipher lists because some duplicates were filtered out.
Unique from previous requests for me:
https://pro-src.com TLSv1.1 ciphers hash: 51f39199142f9474451bf94a47f70a46
Unique from previous requests for you:
https://pro-src.com DEFAULT :!ECDHE+SHA:!AES128-SHA ciphers hash: 59fac59572e9abb2cdf770e0debfdfa3
Those two types of requests are the last two made by the script. I hope that makes sense.
Ah ok, yes. That explains it. Should I generate a report in python2 without the shared ciphers information, or does that render it useless?
If you modify the script, you can still generate a report without the get_ciphers
method for howsmyssl
and ssllabs
.
Also, we can draw conclusion a from your most recent report. The ciphers that are being reported by the socket when using DEFAULT were not unique from the ones reported when using TLSv1.1. I'm guessing that's because of the openssl version and cfscrape from the PR.
Report for python2: https://gist.github.com/lukele/935f48080f56b8221863832debecd320
@lukele Why do you think that none of the hashes matched the python3 report?
Nvm, you're using a different version of openssl for python2. Good results. Now you've got me interested to see the results when using the same version of openssl. :smiley:
Yep, python2 is linked to openssl 1.0.2r
.
Sorry I believed this was known from previous tests. I currently don't have a python2 version linked against openssl 1.1.1
That's gonna take sometime.
Yeh, I forgot. I think we have enough information for now if you don't want to bother with it.
my openssl is OpenSSL 1.1.0i 14 Aug 2018 please help me, How to update openssl to 1.1.1? i have done install from https://slproweb.com/products/Win32OpenSSL.html but, in python not work :(
Thanks for the contributions. I'm no longer getting captchas with the latest revision.
Why do you think it's better to not enable the adapter by default?
And not too important, but to get around the awkward naming, maybe we could remove some of the references to captchas in the code and just call it something like a CloudflareConformingCipherAdapter
, or maybe CloudflareConformingAdapter
? It'd be more generalized; Cloudflare's a pretty innovative and fast-growing company, so I wouldn't be surprised if they throw up hurdles in the future that don't involve captchas.
Why do you think it's better to not enable the adapter by default?
I'm +1 for turning it on by default.
You never know whether or not somebody is gonna attempt to scrape some old/ill configured server which demands the use of the removed TLSv1.0 ciphers. At one point there was an issue with SSLv3 ciphers being removed but that's not longer an issue. So, having it enabled by default would actually be a security feature. Although, it potentially causes request failures where urllib3 would otherwise succeed. It's a highly unlikely scenario though. It's usually the other way around, the client (For example IE6) only has old ciphers and newer server's have dropped those insecure ciphers so a handshake error occurs.
CloudflareConformingAdapter
Naming :thinking:
I agree on all points but I still don't really like it much either. It's the count of words and length that I keep raising an eyebrow at. How about one of these `ConnectionAdapter`, `ContextAdapter`, `SessionAdapter`, `ClientAdapter`, `RequestAdapter`, `TransportAdapter`, `SecureAdapter` or `HTTPSAdapter`?
<BaseAdapter>
-> <HTTPAdapter>
-> <???>
Length doesn‘t bother me, I much prefer to have a name reflecting what it does. CipherCustomizationAdapter
or simply CustomCiphersAdapter
?
The idea of not turning it on by default was to only change the ciphers when absolutely necessary (only initial request to cloudflare would require it, i found in my testing) and stick to urllib3‘s/openssl‘s defaults otherwise.
Not against enabling it by default however.
I'm not opposed to any of the suggestions. ¯\_(ツ)_/¯
Since this is such a small class, there‘s no harm in calling it CustomCiphers
also sounds reasonable in combination with .mount
Ok, let's enable it by default. I can make the change if you guys want to; otherwise please feel free to make the change yourself.
As for the name, I agree with all of the points above. How about just CloudflareAdapter
? Or if you think there's very little chance anyone would ever muck around with it, I'm +1 for CustomCiphers
or CipherOverride
or something like that.
The latest version of cfscrape should not encounter captchas, unless you're using Tor or another IP that Cloudflare has blacklisted. If you're getting a captcha error, first please run
pip install -U cfscrape
and try again. If you're still getting an error, please leave a comment.Please put all captcha challenge-related issues here.
Please run the following to determine the OpenSSL version compiled with your Python binary and include the output in your comment:
(Or
python
instead ofpython3
if running on Python 2.)