HTTPAdapter with SSLContext specified does not use SSLContext's ca_certs on Windows

[matthchr]

My objective was to get requests to use the Windows certificate store rather than the certifi bundle. Maybe this just isn't supported.

I know there is some complexity and has been some debate about how supplying an SSLContext was supposed to work in requests (see #2118). But according to that issue, TransportAdapters (i.e. HTTPAdapter) is the recommended way to provide an SSLContext.

Expected Result

The SSL Context provided would pass its ca_certs along to requests and authentication with a remote endpoint would work.

Actual Result

It didn't work, instead there is a failure looking up the certificate bundle (which I've neglected to deploy alongside my application, so it's not there).

The callstack ends up here:

File "venv\lib\site-packages\requests\adapters.py", line 228, in cert_verify
    "invalid path: {}".format(cert_loc))

Reproduction Steps

import requests
import requests.adapters

# adapted from https://stackoverflow.com/questions/42981429/ssl-failure-on-windows-using-python-requests/50215614
class SSLContextAdapter(requests.adapters.HTTPAdapter):
    def init_poolmanager(self, *args, **kwargs):
        context = ssl.create_default_context()

        kwargs['ssl_context'] = context
        return super(SSLContextAdapter, self).init_poolmanager(*args, **kwargs)

s = requests.Session()
s.mount('https://www.google.com', SSLContextAdapter())
result = s.get('https://www.google.com')

Additionally (as a hack to emulate my enviornment), go rename venv\Lib\site-packages\certifi\cacert.pem

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.4"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1010103f"
  },
  "urllib3": {
    "version": "1.25.7"
  },
  "using_pyopenssl": false
}

[gjb1002]

I agree that this should be much more straightforward than it is. There should be a simple way to tell it to use Windows cert store on Windows, or to hook this behaviour in via an adapter.

I spent quite a while figuring out how to do this, and eventually came up with a solution/workaround.

class WindowsCertStoreAdapter(requests.adapters.HTTPAdapter):
    def cert_verify(self, conn, url, verify, cert):
        super(WindowsCertStoreAdapter, self).cert_verify(conn, url, verify, cert)
        # By default Python requests uses the ca_certs from the certifi module
        # But we want to use the certificate store instead.
        # By clearing the ca_certs variable we force it to fall back on that behaviour 
        conn.ca_certs = None

hooked in in the same way as the description above.

The problem with your code is that it happily loads all the certs from the cert store, but as long as ca_certs is set to point at the certificate bundle, it will load everything from that afterwards, overwriting them. By making sure it's None you keep the windows cert store ones until they're needed. Hope that helps, anyway.

[fedorbirjukov]

pip-system-certs might be of interest for you.

[fedorbirjukov]

@gjb1002, I think the problem with the above code is much simpler. @matthchr just does not load the certificates from the store after creating the context as mentioned on StackOverflow:

context.load_default_certs() # this loads the OS defaults on Windows

[matthchr]

@fedorbirjukov - create_default_context does that automatically, see: https://github.com/python/cpython/blob/477b1b25768945621d466a8b3f0739297a842439/Lib/ssl.py

Moreover the issue is less with the ssl_context and more with the fact that if the certifi bundle is not found requests raises.

I think that @gjb1002's answer will solve this problem, although in my case I just gave in and rewrote our http stack using aiohttp (which does the right thing here and supports cleaner async/await to boot).

[fedorbirjukov]

Now I see. If using this hack, then you should also clear conn.ca_cert_dir = None.

[roelandschoukens]

I have successfully used the solution above before. But it is not available if you use another package that internally uses requests for its HTTP requests. These often have no documented way to access the internal session instance.