feat: ability to override certificate authorities

[redbmk]

Libraries like requests and httpx support environment variables (e.g. REQUESTS_CA_BUNDLE or SSL_CERT_FILE, respectively) to allow for custom certificate authority bundles. I was hoping that SSL_CERT_FILE would work, though that appears to be httpx-specific. Right now, grpclib imports certifi if it exists, and defaults to the system settings only if certifi isn't installed.

It would be cool to have the ability to pass an env to point to a custom CA bundle and/or tell it to use the system certs even if certifi is installed. I've found two workarounds, but neither are ideal.

This one works by loading a specific CA bundle, but it also requires accessing a private field which could potentially change:

channel = Channel(host, int(port), ssl=True)
channel._ssl.load_verify_locations(cafile=os.getenv("SSL_CERT_FILE"))  # type: ignore

This one would use the system defaults, but requires a lot of boilerplate that should probably be updated whenever grpclib changes. This is essentially a copy of _get_default_ssl_context, but removes the cafile:

def _get_ssl_context() -> ssl.SSLContext:
    ctx = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH)
    ctx.minimum_version = ssl.TLSVersion.TLSv1_2
    ctx.set_ciphers("ECDHE+AESGCM:ECDHE+CHACHA20:DHE+AESGCM:DHE+CHACHA20")
    ctx.set_alpn_protocols(["h2"])
    if ssl.HAS_NPN:
        ctx.set_npn_protocols(["h2"])

    return ctx

channel = Channel(host, int(port), ssl=_get_ssl_context())

You could also use the above instead of using channel._ssl, and pass cafile=os.getenv("SSL_CERT_FILE") into create_default_context

[vmagamedov]

ssl=True (loading system certificates managed by OS) and certifi (managing system certificates using Python package manager) support was added mainly to connect to public services with public certificates. The only preferred way to connect to private services with custom certificates is to pass custom SSLContext object explicitly. There are also other options to secure connections (proxies, tunnels).

BTW, SSL_CERT_FILE is not httpx-specific. Looks like SSL_CERT_FILE/-DIR vars are specific to OpenSSL.

>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile=None, capath='/usr/lib/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs')

The only thing that bothers me is that introduction of SSL_CERT_FILE/-DIR in grpclib would be a backward incompatible change because these vars may already be present in some production environments and they may break those configurations.

[redbmk]

Would you be opposed to introducing a new environment variable specific to grpclib that wouldn't introduce any breaking changes? Something like GRPCLIB_CA_BUNDLE maybe, and then fallback to certifi if that's not present, and None otherwise? Or maybe having a public function that builds a context but lets you pass in the context?

I guess I'm not sure how common this use case is, so maybe it's not worth putting in custom code for it. In my case we are using a public endpoint with public certificates, but we have a corporate VPN that intercepts traffic and replaces the certs with ones we "trust" on our systems. We're encouraged to work on VPN as often as possible, but run into issues when specific CA bundles are used instead of the system ones.

We potentially run into issues not just with our own code (where we could create our own context) but with code from other libraries or applications. What we've done that works for the most part is create a custom CA bundle that combines the latest certifi with our internal certs, so that most things work both on and off VPN, as long as we point the right environment variables at it. We've had to do this for requests, httpx, OpenSSL, and node, for example, and it works with grpc, but there's not a simple out-of-box solution for grpclib.

If this doesn't seem like a common enough situation and you're not comfortable with supporting anything other than certifi out of the box, that makes sense to me. But I think ideally it'd be great if we didn't have to recreate the private methods from scratch (that could change over time and we could miss out on potentially important fixes), or access them directly (which could also change over time in a breaking way, but wouldn't be considered a breaking change because it's not a publicly exposed method).

[vmagamedov]

@redbmk Implemented one more way to pass verify locations to a Channel

Now you can use environment variables or explicitly specify location of your certificates.