search

psf / requests

A simple, yet elegant, HTTP library.
https://requests.readthedocs.io/en/latest/
Apache License 2.0
52.17k stars 9.33k forks source link

bad handshake - requests.exceptions.SSLError #3833

Closed filipposantovito closed 7 years ago

filipposantovito commented 7 years ago

Hi,

I read and tried what's in other threads without success. This is the error:

 python -c "import requests;  requests.get('https://acciseonline.agenziadogane.it')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/filippo/src/venvs/test_ssl_sito_dogana/local/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/home/filippo/src/venvs/test_ssl_sito_dogana/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/filippo/src/venvs/test_ssl_sito_dogana/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/filippo/src/venvs/test_ssl_sito_dogana/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/home/filippo/src/venvs/test_ssl_sito_dogana/local/lib/python2.7/site-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')],)",)

it runs inside a virtualenv

(test_ssl_sito_dogana) filippo@savage:~/src/venvs/test_ssl_sito_dogana$ which python
/home/filippo/src/venvs/test_ssl_sito_dogana/bin/python
(test_ssl_sito_dogana) filippo@savage:~/src/venvs/test_ssl_sito_dogana$ pip --version
pip 9.0.1 from /home/filippo/src/venvs/test_ssl_sito_dogana/local/lib/python2.7/site-packages (python 2.7)

here the pip freeze output

(test_ssl_sito_dogana) filippo@savage:~/src/venvs/test_ssl_sito_dogana$ pip freeze | sort
appdirs==1.4.0
backports.shutil-get-terminal-size==1.0.0
cffi==1.9.1
configparser==3.5.0
cryptography==1.7.1
decorator==4.0.11
elpy==1.13.0
enum34==1.1.6
flake8==3.2.1
idna==2.2
importmagic==0.1.7
ipaddress==1.0.18
ipdb==0.10.2
ipython==5.1.0
ipython-genutils==0.1.0
jedi==0.9.0
mccabe==0.5.3
ndg-httpsclient==0.4.2
nose==1.3.7
packaging==16.8
pathlib2==2.2.1
pep8==1.7.0
pexpect==4.2.1
pickleshare==0.7.4
pkg-resources==0.0.0
prompt-toolkit==1.0.9
ptyprocess==0.5.1
pyasn1==0.1.9
pycodestyle==2.2.0
pycparser==2.17
pyflakes==1.5.0
Pygments==2.2.0
pyOpenSSL==16.2.0
pyparsing==2.1.10
requests==2.13.0
scandir==1.4
simplegeneric==0.8.1
six==1.10.0
traitlets==4.3.1
wcwidth==0.1.7

and here some openssl info:

(test_ssl_sito_dogana) filippo@savage:~/src/venvs/test_ssl_sito_dogana$ python -c "import ssl; print ssl.OPENSSL_VERSION"
OpenSSL 1.0.2g  1 Mar 2016

(test_ssl_sito_dogana) filippo@savage:~/src/venvs/test_ssl_sito_dogana$ openssl s_client -connect acciseonline.agenziadogane.it:443 
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = SSL, CN = acciseonline.agenziadogane.it
verify return:1
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=SSL/CN=acciseonline.agenziadogane.it
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFczCCBFugAwIBAgIQc2nmdmmY4REjS+9DUxt9NjANBgkqhkiG9w0BAQsFADCB
kDELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G
[...]
BFDOspLDTFNZ4/WmvfSS/VGZhbqewlpyD+GCYvdiiOrrhxX7cOXkQrBR0ckEypX2
qWl7dMhwlA==
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=SSL/CN=acciseonline.agenziadogane.it
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
---
SSL handshake has read 5585 bytes and written 619 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: FE200000E6B4995A513CC8625648B6F13F32666D5858585812C08958DF770000
    Session-ID-ctx: 
    Master-Key: 35260147D2EEB4B465892E426BBD8504190DDC895FFA894DE0079A40C50247AF1080C1FF792D8E255AF2CD6F2A04087E
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1485422610
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
Lukasa commented 7 years ago

Hi there!

This is not the first time this website has been reported to us as a problem. However, I should note for the future that this website has quite possibly the worst TLS configuration I have ever seen.

Requests refuses to talk to this server in the default configuration because it cannot establish a secure connection: we don't support any of the ciphers that this server does, because all of them are weak. You can get this to work by enabling support for 3DES as shown in this answer, but I strongly recommend you don't do that, and that instead you put pressure on your governmental representatives to fix this terrible TLS configuration.

filipposantovito commented 7 years ago

thank you for your response. I'll try to enable 3DES.

but I strongly recommend you don't do that, and that instead you put pressure on your governmental representatives to fix this terrible TLS configuration.

have you noticed the '.it' at the end of the url? Our government/parliament isn't able to produce a electoral law: it's impossible to 'put pressure' on anyone.... Right now no one cares about IT services... To be honest no one cares about the people, too... it's a sad story....

Lukasa commented 7 years ago

I have, but it's not really my place to make judgements about the Italian electoral system. The best I can really do is point out which organisation needs to take action. =(

filipposantovito commented 7 years ago

the following code works. Thank you for your support. 

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context

# This is the 2.11 Requests cipher string.
CIPHERS = (
    'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
    'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
    '!eNULL:!MD5'
)

class DESAdapter(HTTPAdapter):
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).init_poolmanager(*args, **kwargs)

if __name__ == '__main__':
    s = requests.Session()
    s.mount('https://acciseonline.agenziadogane.it', DESAdapter())
    print s.get('https://acciseonline.agenziadogane.it')
filipposantovito commented 7 years ago

anyway how did you spot that 3des was missing and necessary? I'm asking just to understand the topic and avoid to re-ask for similar problems.

Lukasa commented 7 years ago

So, I have a moderate advantage in that I do maintenance of the TLS configuration in Requests.

However, the Requests cipher string lives at requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS. You can check what that turns out to be on your system in most cases by running

openssl ciphers `python -c "import requests; print(requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS)"`

Of course, that produces a lot of output, so the compressed form used in Requests may be more helpful, which is:

'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5'

This translates to, in order of preference:

What's noticeable about this list? Well, it contains only two stream ciphers: AES in its appropriate stream modes, and ChaCha20. That's because we removed 3DES in the wake of the news that it's insecure for bulk transfer, as we do not control whether or not our users perform bulk transfer so it should be removed. Additionally, 3DES is very slow compared to AES and ChaCha20, so it was usually a sub-optimal choice.

filipposantovito commented 7 years ago

clear. But let me elaborate on the theme.

From https://www.ssllabs.com/ssltest/analyze.html?d=acciseonline.agenziadogane.it&hideResults=on you got TLS_RSA_WITH_3DES_EDE_CBC_SHA is a valid cipher for the site. Now it's obvious that we need to instruct requests/openssl to use that cipher.

The way is to add to the ciphers string the following:

where are these strings listed and documented? I've found a partial answer here: https://wiki.openssl.org/index.php/Manual:Ciphers(1)

looking for TLS_RSA_WITH_3DES_EDE_CBC_SHA in that page you get:

TLS_RSA_WITH_3DES_EDE_CBC_SHA DES-CBC3-SHA

under TLS v1.0 cipher suites.

but how do you know that ECDH+3DES or DH+3DES or RSA+3DES is the correct string to add in order to solve my problem?

Lukasa commented 7 years ago

You don't need to add all three of those: that's just what used to be the Requests cipher string. To add literally just that cipher suite you could add just DES-CBC3-SHA.

However, the others are groups of cipher suites. Again, they use the weird OpenSSL grouping language. "ECDH+3DES" means any cipher suite that uses Elliptic-Curve Diffie-Hellman key exchange and 3DES as the stream cipher. "DH+3DES", same deal: regular Diffie-Hellman and 3DES. Finally, "RSA+3DES" is RSA key exchange and 3DES.

In this case, you can just add the specific string you need (DES-CBC3-SHA). The string you added is a bit more liberal, because it includes forward-secret versions of 3DES that this server doesn't use. That's probably ok, and it's written that way to be a bit more defensive in case other users hit 3DES issues: it basically reverts to the Requests behaviour of 2.11, before we removed 3DES from our supported cipher list.

Unfortunately, the only way to understand what cipher strings do is to understand how a TLS cipher suite is constructed, and to understand the way OpenSSL thinks about them. That's a somewhat complex issue, but let me break it down a bit. BTW, some of this has changed in the upcoming TLSv1.3, so bear that in mind.

In TLSv1.2 and earlier, all cipher suites have got formal IANA-registered names like TLS_RSA_WITH_3DES_EDE_CBC_SHA. These names uniquely identify all the constituent parts of a cipher suite. A cipher suite is made up of the following parts:

The name encodes all of this. For example, consider the following names and their breakdowns:

  1. TLS_RSA_WITH_3DES_EDE_CBC_SHA:
    • Key Exchange: RSA
    • Signing: RSA (this means this must be an RSA certificate). Note that RSA is a unique case in that if it is used as a key exchange mechanism it is also used as the signing mechanism, so only needs to be specified once)
    • Stream Cipher: 3DES in EDE-CBC mode
    • Key Size: Not specified, so it's the default for 3DES, which is 168 bits (3x56)
    • MAC: SHA1

  2. TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

    • Key Exchange: Ephemeral Elliptic-Curve Diffie-Hellman

    • Signing: Elliptic-Curve DSA (not common at the moment)

    • Stream Cipher: AES in GCM mode (the best currently-deployed AES mode)

    • Key Size: 256-bits

    • MAC: Actually, doesn't have one.

      You see, I lied a little bit. Starting in TLSv1.2 we got stream ciphers with AEAD modes. These modes don't require a MAC (or, more properly, the MAC functionality is integrated with the cipher mode). However, these modes do change the way the pseudo-random numbers are generated in TLS, so the part of the name that would normally have the MAC instead have the PRF algorithm. So...

    • PRF: SHA384 (For non-AEAD cipher suites, the PRF is always SHA1 and MD5).

  3. TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
    • Key Exchange: Ephemeral Elliptic-Curve Diffie-Hellman
    • Signing: RSA
    • Stream Cipher: ChaCha20 using Poly1305 authentication
    • Key Size: 256-bits
    • MAC: None, again, ChaCha20/Poly1305 is an AEAD cipher.
    • PRF: SHA256.

As you can see, this is not a totally simple idea. However, the argument goes that you can construct "groups" of cipher suites based on their common functionality. For example, if you like ChaCha20 you could construct a group of cipher suites all based on ChaCha20. This would be any suite that uses ChaCha20 as its stream cipher. In this case, as of TLSv1.2, that list is:

OpenSSL lets you specify these either one-by-one using their full OpenSSL name (e.g. the first one is ECDHE-ECDSA-CHACHA20-POLY1305), or as a group by just using "CHACHA20". If you wanted only the ones that use ECDHE for key exchange, you can join that requirement with a +: "ECDH+CHACHA20". Maybe you want to exclude the PSK ones (often a good idea): in that case, you can say "CHACHA20:!PSK".

The final wrinkle in here is that some of OpenSSL's full names seem a lot shorter than the IANA name, and are missing parts. Consider again our friend TLS_RSA_WITH_3DES_EDE_CBC_SHA. As you noted, OpenSSL's name for this is DES-CBC3-SHA. This only specified the stream cipher, the mode, and the MAC. Why is it missing this other stuff?

The answer is that OpenSSL has defaults: stuff that is assumed. If you don't name the key exchange mechanism, OpenSSL assumes RSA. That means that if you tried to see what OpenSSL uses for "ECDH+3DES", you'll find that DES-CBC3-SHA isn't in there. It's only present for "RSA+3DES".

Essentially the OpenSSL cipher string language is an extremely complex query system. It is moderately difficult to understand, and I recommend playing about with the openssl ciphers command to get a feel for what's going on.

filipposantovito commented 7 years ago

wow. things are a lot clearer now. I think this answer should live on requests' main site.

thank you very much!

Lukasa commented 7 years ago

Thanks!

While I'm glad you liked the answer, however, I don't think Requests should document it. Touching this area of the code is very much like opening the emergency exit on an aircraft in flight. Sure, there might be a good reason to do it, and it should be possible, but the average passenger should never even realise that it can be done, let alone know how.

As much as possible we don't want users to have to touch this code at all. That's why changing this code is tricky (and undocumented). The term I like to use is "attractive nuisance": people who encounter errors when running their code will often start flicking random switches in an attempt to get it to work. verify=False is our one concession to this case in the public API. But we very much don't want the cipher suite list to be a part of this because there are too many bad cipher suites, and the list gets longer over time.

The Requests and urllib3 cipher suite list is maintained by a small number of people who keep a very careful eye on the crypto research and keep it up-to-date. We expand to use new believed-safe ciphers, and we proactively remove ones that are discovered to be bad. That is generally less-likely to happen if users are overriding our choices, so we don't want to make it too easy for users to do that unless they really, really need to.