Buildpack is not adding custom ca-certificates into /etc/ssl/certs/ca-certificates.crt

[benjie91]

Buildpack is not adding custom ca-certificates into /etc/ssl/certs/ca-certificates.crt

Expected Behavior

Custom ca-certificates should be added to the system truststore in /etc/ssl/certs/ca-certificates.crt

Current Behavior

SSL_CERT_DIR is set to /layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates (custom certs location) SSL_CERT_FILE is set to /etc/ssl/certs/ca-certificates.crt (default certs)

Steps to Reproduce

Note: I'm running in an air-gapped network and thus have no use for all the default certs provided

1. Create a python app and call a URL that is signed internally using httpx library.
2. App failed to run because SSL_CERT_FILE does not contain custom certs.
3. The docs of httpx library from Python mentions that it supports SSL_CERT_FILE and SSL_CERT_DIR. I suspects that SSL_CERT_FILE take precedence because once I delete the SSL_CERT_FILE, the call become successful)

[dmikusa]

Thanks for reaching out. A couple of follow up questions.

1. This kind of sounds like perhaps something with the way that the httpx library has implemented things. I like we've been setting SSL_CERT_DIR and SSL_CERT_FILE in this way for quite a while and I know it's been used successfully. As a test, have you tried with a different Python library? requests or one from the stdlib? I'm not suggesting you should change your tools, but it would be helpful to know if this is something specific to httpx or affecting other Python clients as well.

2. Have you tried setting SSL_CERT_FILE to ` (empty string) or an empty or non-existent file? The buildpack should only set this if the env variable is unset, so if you set it to something that might be a way that you can easily get your app working (assuming that your app doesn't like havingSSL_CERT_FILE` set to the container default).

[benjie91]

This was quite some time ago but if I am not wrong, requests library would have worked but not for httpx library. I am using httpx library for async https calls to other APIs.

Looking at my codebase, this is what I have to do for my code to work in production.

if os.getenv("SSL_CERT_FILE"):
  del os.environ["SSL_CERT_FILE"]

As mentioned, once I delete the SSL_CERT_FILE env variable, the httpx call works without any SSL issues. And therefore, if I am right, the SSL_CERT_FILE env variable (i.e. /etc/ssl/certs/ca-certificates.crt) do not contain the custom ca-certificates. I doubt many python library maintainers would have known that there will be cert differences for the env variables SSL_CERT_FILE and SSL_CERT_DIR and thus would have chosen either of the SSL variables.

I am thinking for production container image, it would be good if custom ca-certificates are standardized across both variables.

[dmikusa]

Glad you were able to get your code working!

I am thinking for production container image, it would be good if custom ca-certificates are standardized across both variables.

Unfortunately, that's not possible. The SSL_CERT_FILE env variable points to the set of certificates that ships with the container image. Those are owned by root, and not modifiable by buildpacks.

SSL_CERT_FILE is set to /etc/ssl/certs/ca-certificates.crt (default certs)

The buildpack-added certificates are put into the following location, which is owned by the buildpack user. This is where all the user-added certificates are put.

SSL_CERT_DIR is set to /layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates (custom certs location)

This sounds to me like a library issue. I don't think the OpenSSL docs are very clear, but from reading them I see: When looking up CA certificates for chain building, the OpenSSL library will search for suitable certificates first in CAfile, then in CApath.. To me, that indicates that OpenSSL will first check the trusted certs file, then check the trusted certs directory. It sounds like your library is not doing that. That your library is stopping after it checks the trusted certs file and not checking the directory, which isn't the same as what OpenSSL does.

[benjie91]

I see. This suggestion came because previously I was building my own docker image through dockerfile and I could run commands such as update-ca-certificates after importing the certs to the correct directory. I did not have to do this hackish workaround of modifying my application code directly.

I will try contacting the httpx maintainer library! Thanks for the help.