REQUESTS_CA_BUNDLE overrides `docker.api.client.APIClient` `ca_cert` init parameter

In my current environment requests is used for several different APIs. All of them are under the same ICA and REQUESTS_CA_BUNDLE env var is defined pointing to its cert for proper validation.

Now I've added docker-py to connect to a docker host using TLS. The certificates are issued by a different ICA, only for docker, so I initialize the client as such:

client = docker.DockerClient(
            base_url=f'https://{box.ip}:8080',
            tls=docker.tls.TLSConfig(
                ca_cert='docker.ca.pem',
                client_cert=('client.pem', 'client.key'),
                verify=True,
            ),
        )

After banging for some time trying to understand why I got failed to verify certificate error when using docker CLI (and curl) it worked, I got to this part in requests lib:

https://github.com/psf/requests/blob/master/requests/sessions.py#L710

So, at this point, docker APIClient has self.verify = 'docker.ca.pem (as needed) but the verify kwarg in this merge_environment_settings is None. As it goes into trust_env first, it sets the local verify to REQUESTS_CA_BUNDLE and then when it merges with self.verify it is too late, env var took precedence.

I'll open an issue to requests as, in my opinion, local verify should be start with the kwarg. merged with self.verify and only then with ENV..

For now my existing workarounds are disabling trust_env or subclassing/monkeypatching APIClient to change the base methods (such as _get https://github.com/docker/docker-py/blob/master/docker/api/client.py#L230) to pass self.verify to requests.Session methods (so it actually becomes the "local verify")...

Posting this here more for discussion if there's any proper fix for this that you see without changes to requests

Input on the situation

In Requests, the session_setting loses to the request_setting. The request_setting comes from REQUESTS_CA_BUNDLE. I don't believe this is a Requests library issue given the standard logic in requests.sessions.merge_setting, which is called from merge_environment_settings. It seems pretty clear that since nothing was explicitly passed into the requests call, an environment variable wins out. I personally find it odd that the session loses to an environment variable, so debatable on where fault exists.

IMHO, if this SDK wants to overcome, I suggest we consider how to get the verify to stick. A few ideas come to mind:

Pass the verify (etc.) down in the kwargs through docker.api.client.APIClient.[_post,_get,...] (also what's suggested by @fopina as workaround)
Create a custom Session object to override default behavior (not a good idea IMO)

The call structure I've traced is: docker.api.buid.BuildApiMixin.build (client.api.build) --> docker.api.client.APIClient._post (client.api._post) --> requests.sessions.Session.post --> requests.sessions.Session.request --> requests.sessions.Session.merge_environment_settings --> requests.sessions.merge_settings

This is the error I get on a TLS-required docker host

Adding this section to perhaps help anyone else attempting to find the error details... I only found this issue after I figured out root cause...

When `version` isn't set for DockerClient

On init, the DockerClient makes a call to get the API version

docker.errors.DockerException: Error while fetching server API version: HTTPSConnectionPool(host='localhost', port=2376): Max retries exceeded with url: /version (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))

When `version` is set and a `client.images.build(...)` is called

Setting version avoids the DockerClient init API call to retrieve the version and made it a bit easier to debug the issue.

requests.exceptions.SSLError: HTTPSConnectionPool(host='localhost', port=2376): Max retries exceeded with url: /v1.44/build?t=ubi8-image-name%3A0&q=False&nocache=False&rm=False&forcerm=False&pull=False (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))

Steps to Reproduce

It would be most helpful to have a docker instance requiring TLS to connect. In my situation, docker is a side-car container and not one I setup nor own, so I can't speak to details. Regardless of that, validating that REQUESTS_CA_BUNDLE takes precedence over what's passed in to the docker.tls.TLSConfig.verify can be done without TLS setup.

Here's the code I used to reproduce behavior:

from docker.api import APIClient
from docker import DockerClient
from docker.tls import TLSConfig
from requests.sessions import Session
original_send = Session.send
original_get = APIClient._get
original_post = APIClient._post
import os
os.environ["REQUESTS_CA_BUNDLE"] = "FROM REQUESTS_CA_BUNDLE"

def reproduction(version=None):
    try:
        tls_config = TLSConfig(verify="/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt2")
        client = DockerClient(base_url="https://localhost:2376", tls=tls_config, version=version)
        print(f"Verify passed in to APIClient: {client.api.verify}")
        image_name = "ubi8-image-name:0"
        build_path = "./extensions/molecule/default/ubi8_docker"
        image, build_logs = client.images.build(path=build_path, tag=image_name)
    except:
        return

def hijacked_post_call_from_api_client(self, url, **kwargs):
    """Hijacked to capture on build call"""
    print(f"Kwargs passed to APIClient._post call: {kwargs}")
    original_post(self, url, **kwargs)

def hijacked_get_call_from_api_client(self, url, **kwargs):
    """Hijacked to capture on API version retrieval"""
    print(f"Kwargs passed to APIClient._get call: {kwargs}")
    original_get(self, url, **kwargs)

def hijacked_requests_session_send(self, request, **kwargs):    
    print(f"What requests `send` function receives in kwargs: {kwargs}")
    original_send(self, request **kwargs)

APIClient._post = hijacked_post_call_from_api_client
APIClient._get = hijacked_get_call_from_api_client
Session.send = hijacked_requests_session_send

print(f"{'-'*60}")
print("Creating DockerClient where it will make a `get` call to get the API version:")
print(f"{'-'*60}")
reproduction()
print(f"{'-'*60}")
print("Creating DockerClient where it will make a `post` call to do a build:")
print(f"{'-'*60}")
reproduction(version="1.44")

docker / docker-py