Trouble with the packaging tutorial: Hostname mismatch with provided self-signed certificate

[mpeter50]

OS version

Windows 10

Python version

3.10.11

Pip version

23.1.2

Guide link

https://packaging.python.org/tutorials/packaging-projects

Problem description

I wanted to upload the test package with twine to my package registry, that runs on the local network. It is a Gitea server, for which I use a wildcard self-signed certificate.

When I pass the certificate's PEM file (obtained from my web browser, which communicates with the server without issue, after accepting the cert once) to the twine command, it repeats this error while retrying a few times: "SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'REDACTED'"

When the error happens, I use this command:

py -m twine upload --cert "<path-to-pem-file>" --repository-url https://REDACTED/api/packages/mpeter/pypi dist/*

Error message

Uploading distributions to https://REDACTED/api/packages/mpeter/pypi
Enter your username: MPeter
Enter your password: 
Uploading example_package_mpeter-0.0.1-py3-none-any.whl
WARNING  Retrying (Retry(total=9, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,   
         certificate is not valid for 'REDACTED'. (_ssl.c:1007)"))': /api/packages/mpeter/pypi
WARNING  Retrying (Retry(total=8, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,   
         certificate is not valid for 'REDACTED'. (_ssl.c:1007)"))': /api/packages/mpeter/pypi
WARNING  Retrying (Retry(total=7, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,   
         certificate is not valid for 'REDACTED'. (_ssl.c:1007)"))': /api/packages/mpeter/pypi
WARNING  Retrying (Retry(total=6, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,   
         certificate is not valid for 'REDACTED'. (_ssl.c:1007)"))': /api/packages/mpeter/pypi
WARNING  Retrying (Retry(total=5, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,   
         certificate is not valid for 'REDACTED'. (_ssl.c:1007)"))': /api/packages/mpeter/pypi
WARNING  Retrying (Retry(total=4, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,   
         certificate is not valid for 'REDACTED'. (_ssl.c:1007)"))': /api/packages/mpeter/pypi
WARNING  Retrying (Retry(total=3, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,   
         certificate is not valid for 'REDACTED'. (_ssl.c:1007)"))': /api/packages/mpeter/pypi
WARNING  Retrying (Retry(total=2, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,   
         certificate is not valid for 'REDACTED'. (_ssl.c:1007)"))': /api/packages/mpeter/pypi
WARNING  Retrying (Retry(total=1, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,   
         certificate is not valid for 'REDACTED'. (_ssl.c:1007)"))': /api/packages/mpeter/pypi
WARNING  Retrying (Retry(total=0, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch,   
         certificate is not valid for 'REDACTED'. (_ssl.c:1007)"))': /api/packages/mpeter/pypi
  0% ---------------------------------------- 0.0/17.8 kB • --:-- • ?
Traceback (most recent call last):
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\connection.py", line 635, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\connection.py", line 776, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\util\ssl_.py", line 466, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\util\ssl_.py", line 510, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "<redacted>\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "<redacted>\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1071, in _create
    self.do_handshake()
  File "<redacted>\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'REDACTED'. (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'REDACTED'. (_ssl.c:1007)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\requests\adapters.py", line 486, in send
    resp = conn.urlopen(
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\connectionpool.py", line 874, in urlopen
    return self.urlopen(
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\connectionpool.py", line 874, in urlopen
    return self.urlopen(
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\connectionpool.py", line 874, in urlopen
    return self.urlopen(
  [Previous line repeated 7 more times]
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\urllib3\util\retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='REDACTED', port=443): Max retries exceeded with url: /api/packages/mpeter/pypi (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certif
icate verify failed: Hostname mismatch, certificate is not valid for 'REDACTED'. (_ssl.c:1007)")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<redacted>\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "<redacted>\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\twine\__main__.py", line 51, in <module>
    sys.exit(main())
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\twine\__main__.py", line 33, in main
    error = cli.dispatch(sys.argv[1:])
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\twine\cli.py", line 123, in dispatch
    return main(args.args)
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\twine\commands\upload.py", line 198, in main
    return upload(upload_settings, parsed_args.dists)
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\twine\commands\upload.py", line 142, in upload
    resp = repository.upload(package)
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\twine\repository.py", line 186, in upload
    resp = self._upload(package)
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\twine\repository.py", line 172, in _upload
    resp = self.session.post(
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\requests\sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "<redacted>\python_packaging_tutorial\_venv\lib\site-packages\requests\adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='REDACTED', port=443): Max retries exceeded with url: /api/packages/mpeter/pypi (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificat
e verify failed: Hostname mismatch, certificate is not valid for 'REDACTED'. (_ssl.c:1007)")))

[mpeter50]

I wanted to figure out how is the failing check implemented. So far I have found by setting breakpoints and doing a regular requests.get() to this domain that the exception is coming from inside ssl.SSLSocket.do_handshake, from the call to self._sslobj.do_handshake().

Unfortunately, the definition of that function cannot be found by my IDE, and trying to step into it results in ending up in the next finally block, which probably means that this is native code, which I cannot inspect.

[mpeter50]

The problem was that my self-signed certificate was incorrect. And not specifically with twine, but generally python's requests pacakge did not like it.

The reason is that for ~23 years now, the Common Name of a certificate cannot be used for the domain name (you can put it there for your convenience, but software wont look there for it). Details here: 1, 2. Instead, the domain name has to be specified in the Subject Alt Names list.

If someone reading this also wants to upload their package to a selfhosted package index, but has also received this error message, you should make sure that your certificate is correct regarding that. This solution helped a lot in figuring out how to make a correct TLS certificate, but for self signed certs you will have to use -x509 instead of -new and -out cert.crt instead of -out cert.csr in the command arguments, and also x509_extensions = req_ext instead of req_extensions = req_ext in the config file. In the config file you may also want to use basicConstraints=CA:TRUE, pathlen: 0 in the req_ext INI section.