psf / requests

A simple, yet elegant, HTTP library.
https://requests.readthedocs.io/en/latest/
Apache License 2.0
52.17k stars 9.33k forks source link

OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF') #4294

Closed thomaswpp closed 6 years ago

thomaswpp commented 7 years ago

Summary. I'm on Mint 18.01 and I try requests in site https://www.sifge.caixa.gov.br/Cidadao/Crf/FgeCfSCriteriosPesquisa.asp, but not function.. I did download certificate in the firefox and then I add certificate in /etc/ssl/cert with comand

sudo mkdir /usr/share/ca-certificates/extra
sudo cp WWWSIFGECAIXAGOVBR.crt /usr/share/ca-certificates/extra/WWWSIFGECAIXAGOVBR.crt
sudo dpkg-reconfigure ca-certificates

And then I add the variable at the end of the file ~/.bashrc

REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt # ca-bundle.crt
export REQUESTS_CA_BUNDLE 

And I executed command

source .bashrc

Actual Result

During handling of the above exception, another exception occurred:

TrTraceback (most recent call last):
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 441, in wrap_socket
    cnx.do_handshake()
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1716, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1449, in _raise_ssl_error
    raise SysCallError(-1, "Unexpected EOF")
OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 448, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.sifge.caixa.gov.br', port=443): Max retries exceeded with url: /Cidadao/Crf/FgeCfSCriteriosPesquisa.asp?ImportWorkEmpregadorCodigoInscricaoAlfanum=10778237%2F0001-51&ImportURL=..%2FCrf%2FFgeCfSConsultaRegularidade.asp&tipoinscricao=1&ImportEstadoSigla=&resultadopath5=5050-5045-5105-5111-5102&resultadopath3=5049-5045-5105-5111-5102&sltCidade=Produtos%2Be%2BServi%25E7os&resultadopath=5050-5045-5105-5111-5102&ImportWorkEmpregadorTipoInscricao=1&resultadopath4=5053-5045-5105-5111-5102&resultadopath2=5052-5045-5105-5111-5102&navegue=Navegue%2Bpela%2BCAIXA&txtConsulta=35263 (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "crawler.py", line 118, in <module>
    r = get_dados()
  File "crawler.py", line 109, in get_dados
    r = s.get(url_get, headers=HEADERS, params=payload, verify=True, allow_redirects=False);
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/thomas/workspace/prime/caixa/venv/lib/python3.5/site-packages/requests/adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.sifge.caixa.gov.br', port=443): Max retries exceeded with url: /Cidadao/Crf/FgeCfSCriteriosPesquisa.asp?ImportWorkEmpregadorCodigoInscricaoAlfanum=10778237%2F0001-51&ImportURL=..%2FCrf%2FFgeCfSConsultaRegularidade.asp&tipoinscricao=1&ImportEstadoSigla=&resultadopath5=5050-5045-5105-5111-5102&resultadopath3=5049-5045-5105-5111-5102&sltCidade=Produtos%2Be%2BServi%25E7os&resultadopath=5050-5045-5105-5111-5102&ImportWorkEmpregadorTipoInscricao=1&resultadopath4=5053-5045-5105-5111-5102&resultadopath2=5052-5045-5105-5111-5102&navegue=Navegue%2Bpela%2BCAIXA&txtConsulta=35263 (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))

Reproduction Steps

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context

# This is the 2.11 Requests cipher string, containing 3DES.
CIPHERS = (
 'DES-CBC-SHA:DES-CBC3-SHA:'
  'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
   'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
   '!eNULL:!MD5'
)

class DESAdapter(HTTPAdapter):
    """
    A TransportAdapter that re-enables 3DES support in Requests.
    """
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).init_poolmanager(*args, **kwargs)

    def proxy_manager_for(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).proxy_manager_for(*args, **kwargs)

HEADERS = {
    'Host':'www.sifge.caixa.gov.br',
    'User-Agent':'Mozilla/5.0 (X11; Ubuntu; Linu…) Gecko/20100101 Firefox/55.0',
    'Accept':'text/html,application/xhtml+xm…plication/xml;q=0.9,*/*;q=0.8',
    'Accept-Language':'en-US,en;q=0.5',
    'Accept-Encoding':'gzip, deflate, br',
    'Content-Type':'application/x-www-form-urlencoded',
    'Content-Length':'460',
    'Referer':'https://www.sifge.caixa.gov.br…f/FgeCfSCriteriosPesquisa.asp',
    'Cookie':'ASPSESSIONIDQSBASDRB=AMFJMBDCJGFKCOKJPKODJIED',
    'Connection':'keep-alive',
    'Upgrade-Insecure-Requests':'1'
}

url_get = 'https://www.sifge.caixa.gov.br/Cidadao/Crf/FgeCfSCriteriosPesquisa.asp'
payload = {
        'ImportWorkEmpregadorTipoInscricao':'1',
        'ImportURL':'../Crf/FgeCfSConsultaRegularidade.asp',
        'sltCidade':'Produtos+e+Servi%E7os',
        'navegue':'Navegue+pela+CAIXA',
        'tipoinscricao':'1',
        'ImportWorkEmpregadorCodigoInscricaoAlfanum':'10778237/0001-51',
        'ImportEstadoSigla':'',
        'resultadopath':'5050-5045-5105-5111-5102',
        'resultadopath2':'5052-5045-5105-5111-5102',
        'resultadopath3':'5049-5045-5105-5111-5102',
        'resultadopath4':'5053-5045-5105-5111-5102',
        'resultadopath5':'5050-5045-5105-5111-5102',
        'txtConsulta':'35263'
    }
s = requests.Session()
s.mount('https://www.sifge.caixa.gov.br', DESAdapter())
r = s.get(url_get, headers=HEADERS, params=payload, verify=True);

System Information

$ python --version
Python 3.5.2
$ openssl version
OpenSSL 1.0.2g  1 Mar 2016
Linux Mint 18.1
$ pip freeze
argh==0.26.2
asn1crypto==0.22.0
beautifulsoup4==4.6.0
blinker==1.4
brotlipy==0.6.0
bs4==0.0.1
certifi==2017.7.27.1
cffi==1.11.0
chardet==3.0.4
click==6.7
construct==2.8.14
cryptography==2.0.3
cssutils==1.0.2
EditorConfig==0.12.1
html2text==2016.9.19
hyperframe==4.0.2
idna==2.6
jsbeautifier==1.6.14
lxml==3.8.0
ndg-httpsclient==0.4.3
packaging==16.8
passlib==1.7.1
pathtools==0.1.2
pkg-resources==0.0.0
pyasn1==0.3.5
pycparser==2.18
pyOpenSSL==17.3.0
pyparsing==2.2.0
PyYAML==3.12
requests==2.18.4
six==1.10.0
sortedcontainers==1.5.7
tornado==4.4.3
urllib3==1.22
watchdog==0.8.3
$ openssl s_client -tls1 -connect www.sifge.caixa.gov.br:443 -showcerts

CONNECTED(00000003)
depth=0 C = BR, O = ICP-Brasil, OU = Caixa Economica Federal, OU = AC CAIXA PJ SSL v2, CN = WWW.SIFGE.CAIXA.GOV.BR
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = BR, O = ICP-Brasil, OU = Caixa Economica Federal, OU = AC CAIXA PJ SSL v2, CN = WWW.SIFGE.CAIXA.GOV.BR
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=BR/O=ICP-Brasil/OU=Caixa Economica Federal/OU=AC CAIXA PJ SSL v2/CN=WWW.SIFGE.CAIXA.GOV.BR
   i:/C=BR/O=ICP-Brasil/OU=Caixa Economica Federal/CN=AC CAIXA PJ SSL v2
-----BEGIN CERTIFICATE-----
MIIGyjCCBLKgAwIBAgIITmChX/KLNhAwDQYJKoZIhvcNAQELBQAwYTELMAkGA1UE
BhMCQlIxEzARBgNVBAoMCklDUC1CcmFzaWwxIDAeBgNVBAsMF0NhaXhhIEVjb25v
bWljYSBGZWRlcmFsMRswGQYDVQQDDBJBQyBDQUlYQSBQSiBTU0wgdjIwHhcNMTcw
NDA3MTI0NDExWhcNMTgwNDA3MTI0NDExWjCBgjELMAkGA1UEBhMCQlIxEzARBgNV
BAoMCklDUC1CcmFzaWwxIDAeBgNVBAsMF0NhaXhhIEVjb25vbWljYSBGZWRlcmFs
MRswGQYDVQQLDBJBQyBDQUlYQSBQSiBTU0wgdjIxHzAdBgNVBAMMFldXVy5TSUZH
RS5DQUlYQS5HT1YuQlIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDI
vvdrLVd1yZVu1Y8ewTZN7GA5OgmsrciO5oLMi+UtfmxbA5ltYKcJGIPJZFglNUwy
ZbW/jTofRxlqjP9xPaOujm6o0rKsP2TzrLM/4ZhVnUmOeUl8ZC9M8S0qfOPNiFj8
lSzq9E7fbpiOX38+WfM9V6h0MAkoGCmM7kw7SkiwmsaIV37WVjWtjq8soqwALLcB
EXlXKB0FJbVs3vuh1CYGzJ2qNrXYZsuChgVS7tXxgE9ndRBqrOh/i2Kp1MITCRPi
ePmkl8aiFhfSNU9ey/cO8liJ231JEM+hQK2CuMxWjlmFSPLHAUAB1sxxo1zhOtEI
CsBuzeQbtzLVfYKpKGUnAgMBAAGjggJiMIICXjAOBgNVHQ8BAf8EBAMCBeAwHQYD
VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBSWCqZPEwyi/9hN
mbpXGmDIXtZmNjAfBgNVHSMEGDAWgBTz1fJRwuxpPdoF61ax83GCrCuC5DCBtQYD
VR0RBIGtMIGqoCIGBWBMAQMIoBkEF0NBSVhBIEVDT05PTUlDQSBGRURFUkFMoEAG
BWBMAQMEoDcENTI1MDUxOTY5MDAwMTg4ODg3MDQxNzAwNjU3MDM3NzAwMDAwMDYz
NDk1NTUzNERFVFJBTlJKoBkGBWBMAQMDoBAEDjAwMzYwMzA1MDAwMTA0oCcGBWBM
AQMCoB4EHFJJQ0FSRE8gUEFTQ0hPRVRPIERPUyBTQU5UT1MwagYDVR0gBGMwYTBf
BgdgTAECAYFKMFQwUgYIKwYBBQUHAgEWRmh0dHA6Ly9jZXJ0aWZpY2Fkb2RpZ2l0
YWwuY2FpeGEuZ292LmJyL2RvY3VtZW50b3MvZHBjYWMtY2FpeGFwanNzbC5wZGYw
bgYDVR0fBGcwZTAwoC6gLIYqaHR0cDovL2xjci5jYWl4YS5nb3YuYnIvYWNjYWl4
YXBqc3NsdjIuY3JsMDGgL6AthitodHRwOi8vbGNyMi5jYWl4YS5nb3YuYnIvYWNj
YWl4YXBqc3NsdjIuY3JsMFkGCCsGAQUFBwEBBE0wSzBJBggrBgEFBQcwAoY9aHR0
cDovL2NlcnRpZmljYWRvZGlnaXRhbC5jYWl4YS5nb3YuYnIvYWlhL2FjY2FpeGFw
anNzbHYyLnA3YjANBgkqhkiG9w0BAQsFAAOCAgEAqziOisz/gHE2Yc8pMRQ9sUjJ
5t7xn0T92lrJlJkeIy6WKqOBUAN9SpAs8O/9QHZGIDJ5yUnUnOjgluH+3RolW7/9
iJDqgv8MI23nKNBPoOedDBE5nSQqVRplEefpjCVFpgoWmzhyLcnz2A3bY4T0qYYj
BsVmOk1xpILAqzaimUhow9JofYolBwHLGJjD3qfzLh29S19GFTuD9MtXroHta/jh
/6PEmKQ7JEitClpx/NBZVukDffh2hBgZ6pDSEltG0vF2gypoezvKCmAYikdLDDim
ALyqM5ZRQ+EJz++5ER4iO9h/NMIMlgm7n7Oq9GD8N4dQ+KqUGWETAjQeRS65aNJG
PIOBmqa5O9Un6XmYR3TXutzdzcI+2ZVUKhnz2SqXVITdAB8XfEbVG4L/DmMzO1FH
0HEmgv53NW9rMqN6iEmeucEP3hl9oZNp0f7/P2jLx6xnpPLXRViPVxfJLZU5fA14
ytS4TTRBI0U4qAJkOQS/O/D6hLpdKlpubrjYR7nzsEy7/W8dbRGmQ3RRhze9iAQ5
pb/p79WmKArJvIm4N0wfFBCxQ35XTjVhls/BT+dEV/uIqfQTu3nZPA7bkIxkpfao
Adq1x9U8rIcgYoNpaGuYyGzYwJddKS828NEJKFT48SkzJNrhJ/P3QOYArqOCnBBg
7HSm9fg7r1DVh0sOeT8=
-----END CERTIFICATE-----
---
Server certificate
subject=/C=BR/O=ICP-Brasil/OU=Caixa Economica Federal/OU=AC CAIXA PJ SSL v2/CN=WWW.SIFGE.CAIXA.GOV.BR
issuer=/C=BR/O=ICP-Brasil/OU=Caixa Economica Federal/CN=AC CAIXA PJ SSL v2
---
No client certificate CA names sent
---
SSL handshake has read 1885 bytes and written 507 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-MD5
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-MD5
    Session-ID: 6F0C0000A0F84D55DBE77861EE7B259F0CE087027969084185C6EFCC96D653EC
    Session-ID-ctx: 
    Master-Key: 34CA4185EAFBACA9D52E27B6C796FADB5C0A9CB1586257C7FCD6F40F74E5C04BEA48C257B71010A3F4AFC301673E32E3
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1505634687
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
Lukasa commented 7 years ago

With modern OpenSSL defaults it turns out that this server shuts the connection down. Likely this is because the remote server is getting confused by us advertising support for TLSv1.1 and 1.2. You could try changing your context = create_urllib3_context(ciphers=CIPHERS) line to:

context =create_urllib3_context(ciphers=CIPHERS) 
context.options |= ssl.OP_NO_TLSv1_2 | ssl.OP_NO_TLSv1_1

that might help.

thomaswpp commented 7 years ago

I tried, but it's the same mistake. I've tried everything, but my knowledge is limited and I do not know what I can try more.

Lukasa commented 7 years ago

It's very hard to know what's going on here: this server's configuration is so bad that it may simply not be possible to communicate with it using your copy of OpenSSL.

thomaswpp commented 7 years ago

Thanks for everything, you're right, the server is really bad.

epleterte commented 6 years ago

I got the same error message (SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",)), but after doing pip install ndg-httpsclient my troubles went away.

amirouche commented 6 years ago

@epleterte how just installing a module can help? You do not import it at some point?

sigmavirus24 commented 6 years ago

@amirouche Requests detects the presence of certain modules and uses them to enhance security on old and insecure versions of Python.