SSL misconfigured, cannot find certs

[layday]

Using the pre-built binary from the macOS template, I am unable to make SSL connections. This appears to be because the SSL cert paths are misconfigured. With a briefcase app set up to load a __main__.py containing:

import ssl

print(ssl.get_default_verify_paths())

Outputs:

DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/Users/runner/work/Python-Apple-support/Python-Apple-support/build/macOS/openssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/Users/runner/work/Python-Apple-support/Python-Apple-support/build/macOS/openssl/certs')

Attempting to make an SSL connection:

import socket

context = ssl.create_default_context()

with socket.create_connection(('beeware.org', 443)) as sock:
    with context.wrap_socket(sock, server_hostname='beeware.org') as ssock:
        ssock.send(b'HEAD / HTTP/1.0\r\nHost: beeware.org\r\n\r\n')

Raises:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

[freakboy3742]

Thanks for the report (and apologies for the delay in responding).

This is actually a problem of usage, rather than a bug in the SSL configuration. Explaining why is a little complex, though.

tl;dr - the fix is to use a package like certifi. This is a Python package that provides a maintained root certificate that can be installed as a normal pip package:

pip install certifi

(and/or added to your briefcase requirements); then, in your code, when you create the SSL context, pass in the location of the CAfile:

import certifi
context = ssl.create_default_context(cafile=certifi.where())

The longer answer:

SSL verification is based on verifying a certificate chain, back to a root certificate. OpenSSL doesn't provide this root certificate itself - it's usually considered the domain of the operating system. macOS doesn't use OpenSSL natively; but when you install OpenSSL via homebrew (or similar), one of the steps of the installation process (either explicitly or implicitly) is to convert the system root keystore into an OpenSSL .pem file that can be used as a root certificate.

In the absence of a root certificate, OpenSSL will attempt to use a local issuer certificate - which causes the error that you've reported. The file location that it references is a location that is defined during the build process, but isn't actually useful anywhere else.

So - why not just include a root certificate as part of Python-Apple-support? Well, firstly, it would make this project part of the critical security chain of projects using SSL. If a root certificate ever needed to be updated or revoked, this project would become responsible for propagating that update to users. That's a collection of headaches I'd rather avoid - for the same reason the OpenSSL project avoids them :-)

Even if we did want to wear this headache, the approach taken by Python-Apple-support that allows for apps to be relocatable complicates the process. On a normal UNIX system, OpenSSL is installed in a known system location like /usr/local/openssl, and so the build process can hard-code the location of this root certificate based off this location. However, apps built with this support package are relocatable by design - which means they can't hard-code a single location on disk where their certificates are stored.

So why not use the root certificate provided by macOS and iOS? Well, it isn't in .pem format, so it isn't compatible with OpenSSL. It can be converted... but then the converted version would need to be stored somewhere, and we're back to the same problem.

The good news is that SSL is extremely configurable - either using the cafile and capath arguments when creating contexts, or by setting the SSL_CERT_DIR and SSL_CERT_FILE environment variables. So, by (a) sourcing a third party root certificate, and (b) explicitly telling SSL operations which SSL certificate file to use as a root, you can open an SSL connection, and have confidence that the connection is secure.

[layday]

I'm not against using certifi (and I have been doing just that) but unless something has changed recently, OpenSSL (LibreSSL, rather) is part of the standard macOS installation and cert.pem can be found in /etc/ssl:

~ $ command -v openssl
/usr/bin/openssl
~ $ openssl version
LibreSSL 2.8.3
~ $ ls /etc/ssl
cert.pem    certs       openssl.cnf x509v3.cnf

(Thanks for the detailed answer!)

[freakboy3742]

Huh... well that's interesting - I knew macOS had adopted LibreSSL, but I assumed it was in a modified form that was reading from a different certificate store.

I've done a quick test, and at the very least, you can pass in cafile=/etc/ssl/cert.pem when creating the context; and if you build the support package configuring the ssl directory as /etc/ssl, the default context works.

I need to poke around to see if there's a comparable analog on iOS/tvOS/watchOS - but even if there isn't, this looks like it might be a viable solution for macOS. Thanks for the tip!

[freakboy3742]

...and it looks like iOS uses /etc/ssl as well. I'll make the configuration change, and roll this into the next support package release. In the meantime, the manual configuration I mentioned earlier (either /etc/ssl or the certifi store) will work.