search

psf / requests

A simple, yet elegant, HTTP library.
https://requests.readthedocs.io/en/latest/
Apache License 2.0
52.09k stars 9.31k forks source link

Misleading certificate error if HTTPS URL domain does not comply to RFC 1035 #6518

Closed volkerjaenisch closed 1 year ago

volkerjaenisch commented 1 year ago

Dear Python requests people!

Please be not annoyed by this Bug report. I know it is not your bug, in fact it is no bug at all. But it is a pitfall a mile deep and myself has fallen twice. And unfortunately it pops up using your lib. So I wrote this bug report mainly as a pointer for others stumbling over the same stone in the SSL-Lib/RFC1035 and attributing an error to you.

Probably there is a way in requests to make a better user experience like testing for RFC 1035 compliance before calling the SSL-code.

To reproduce:

Given a wildcard certificate for *.dev.inqbus.de and a webserver serving the two subdomains utilizing this certificate:

a= request(url='https://backend.dev.inqbus.de', method='GET')
b= request(url='https://test_backend.dev.inqbus.de', method='GET')

Expected Result

Both commands succeed

Actual Result

a= request(url='https://backend.dev.inqbus.de', method='GET')
a= request(url='https://test_backend.dev.inqbus.de', method='GET')
Traceback (most recent call last):
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 783, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 469, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 513, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'test_backend.dev.inqbus.de'. (_ssl.c:1123)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'test_backend.dev.inqbus.de'. (_ssl.c:1123)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='test_backend.dev.inqbus.de', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'test_backend.dev.inqbus.de'. (_ssl.c:1123)")))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/volker/workspace/PROGRAMS/pycharm-2022.3/plugins/python/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/volker/workspace/venvs/vtt-interface-backend-Fy3roxN1-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='test_backend.dev.inqbus.de', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'test_backend.dev.inqbus.de'. (_ssl.c:1123)")))

The first call runs smoothly. The second calls produces the above trace. The error cause in the trace (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'test_backend.dev.inqbus.de'. (_ssl.c:1123)")))

The stack trace in the not RFC 1035 case is misleading.

Reproduction Steps

Please have a look above

System Information

$ python -m requests.help
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.2.0"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.4"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.2"
  },
  "platform": {
    "release": "5.10.0-24-amd64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "101010ef"
  },
  "urllib3": {
    "version": "2.0.4"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}
sigmavirus24 commented 1 year ago

Hi @volkerjaenisch

The error message is coming from several layers below us and as a rule we don't stifle exception details.

The error message that is confusing is from the Python standard ssl library.

Furthermore, it's confusing but not misleading. For the clarification for others who I'm certain will find this and think they have the same issue without understanding fully - the issue here is that the _ in your subdomain is not a valid domain name as _ is not allowed there. So something about your set up is very off.

The error message as I said comes from ssl and probably more accurately from OpenSSL. This we have no good way to clarify as we don't have any more details than what we show you.