search

psf / requests

A simple, yet elegant, HTTP library.
https://requests.readthedocs.io/en/latest/
Apache License 2.0
51.6k stars 9.23k forks source link

Failed to ignore the SSL certificate verification when using `verify=False` option #6669

Closed urbanogilson closed 2 months ago

urbanogilson commented 2 months ago

Requests is not ignoring the SSL certificate verification when using the verify=False option.

Expected Result

Same as curl with -k

$ curl -k https://website
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.20.1</center>
</body>
</html>

or

import httpx
session = httpx.Client(verify=False)
session.get("https://website")

Actual Result

>> import requests
>> requests.get("https://website", verify=False)

Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connection.py", line 635, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connection.py", line 776, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 466, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 510, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='website', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vscode/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/vscode/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/vscode/.local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/vscode/.local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/vscode/.local/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='website', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)')))

The result is similar to curl without -k.

$ curl -v https://website
*   Trying 255.255.255.255:443...
* Connected to website (255.255.255.255) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Reproduction Steps

import requests
session = requests.Session()
session.verify = False
session.get("https://website")

or 

import requests
session.get("https://website", verify=False)

System Information

Tested in two systems

$ python -m requests.help
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.1.0"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.4"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.12"
  },
  "platform": {
    "release": "5.15.146.1-microsoft-standard-WSL2",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "1010117f"
  },
  "urllib3": {
    "version": "2.0.3"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}
$ python -m requests.help
{
  "chardet": {
    "version": "5.2.0"
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": "42.0.5"
  },
  "idna": {
    "version": "3.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.12"
  },
  "platform": {
    "release": "6.1.58+",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "30200010",
    "version": "24.1.0"
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "30000020"
  },
  "urllib3": {
    "version": "2.0.7"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": true
}

Note: URL and IP were masked to https://website and 255.255.255.255 respectively

sigmavirus24 commented 2 months ago

A handshake error is not the same as verifying a trust chain. My guess is that requests/urllib3 and your server cannot agree on ciphersuites to use during the handshake. Otherwise you would see a very different exception message.

urbanogilson commented 2 months ago

From https://www.ssllabs.com/ssltest the supported cipher suites:

Protocols
TLS 1.3 No
TLS 1.2 Yes
TLS 1.1 Yes
TLS 1.0 Yes
SSL 3   No
SSL 2   No

Cipher Suites
# TLS 1.2 (suites in server-preferred order)
TLS_RSA_WITH_AES_256_CBC_SHA256 (0x3d)   WEAK   256
TLS_RSA_WITH_AES_256_CBC_SHA (0x35)   WEAK  256
TLS_RSA_WITH_AES_256_GCM_SHA384 (0x9d)   WEAK   256
TLS_RSA_WITH_AES_128_CBC_SHA256 (0x3c)   WEAK   128
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)   WEAK  128
TLS_RSA_WITH_AES_128_GCM_SHA256 (0x9c)   WEAK   128
# TLS 1.1 (suites in server-preferred order)
TLS_RSA_WITH_AES_256_CBC_SHA (0x35)   WEAK  256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)   WEAK  128
# TLS 1.0 (suites in server-preferred order)
TLS_RSA_WITH_AES_256_CBC_SHA (0x35)   WEAK  256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)   WEAK  128
sigmavirus24 commented 2 months ago

Yeah, the only cipher suites this site offers are awful, and I suspect that if you do the following:

from urllib3.util.ssl_ import create_urllib3_context

for item in create_urllib3_context().get_ciphers():
    print(item['name'])

I would suggest trying the following:

import urllib3

http = urllib3.PoolManager()
response = http.request('GET', 'https://...', ca_certs="CERT_NONE")

I expect you'll encounter the same issue.

urbanogilson commented 2 months ago

You are right, it is reproducible using urllib3

from urllib3.util.ssl_ import create_urllib3_context

for item in create_urllib3_context().get_ciphers():
    print(item['name'])

TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_GCM_SHA256
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA256
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES256-SHA256
DHE-RSA-AES128-SHA256
http = urllib3.PoolManager(ca_certs=None)
response = http.request('GET', 'https://website')

Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connection.py", line 635, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connection.py", line 776, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 466, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 510, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/_request_methods.py", line 110, in request
    return self.request_encode_url(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/_request_methods.py", line 143, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 874, in urlopen
    return self.urlopen(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 874, in urlopen
    return self.urlopen(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 874, in urlopen
    return self.urlopen(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/home/vscode/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='websitw', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)')))
sigmavirus24 commented 2 months ago

Yeah and as you can see the ciphers that are default on your system (because the version of urllib3 you're using relies on system defaults) don't overlap with the very weak set offered by your server. You can customize the ciphers in creating the context for urllib3 and create a custom requests adapter to work around this but this is not a bug. Using the default ciphers is a good security practice and we will not be modifying things to make default ciphers accept weaker algorithms