Feature: config option for `pro` to explicitly trust a cert that is not in the system trust store

[nishant-dash]

Edit: See https://github.com/canonical/ubuntu-pro-client/issues/3284#issuecomment-2371593601 for a summary as of 2024-09-24

Description of the bug

Ubuntu Pro Client can not talk to a https contracting server that uses self signed certificates. It fails on verifying the cert. chain.

$ ubuntu-advantage status --all --format json | jq .
{
  "environment_vars": [],
  "errors": [
    {
      "message": "Failed to access URL: https://REDACTED/v1/resources?architecture=amd64&kernel=5.15.0-1060-kvm&series=jammy&virt=qemu\nCannot verify certificate of server\nPlease check your openssl configuration.",
      "message_code": "ssl-verification-error-openssl-config",
      "service": null,
      "type": "system"
    }
  ],
  "result": "failure",
  "services": [],
  "warnings": []
}

This happens even if you add the certificate chain to the VM's trust store. On the other hand, curl works just fine. Looking at https://github.com/canonical/ubuntu-pro-client/blob/cd40f70fed44abd46dbc7f7679f384348e85ac06/uaclient/http/__init__.py#L163-L172 and following the urllib and sslcontext logic, this creates a Default context with ssl which DOES NOT load the ca certs on the system.

However, if you create a context pointing the right file or dir such as

ctx = ssl.create_default_context(cafile="/usr/lib/ssl/certs/ca-certificates.crt")

and pass it to

resp = request.urlopen(req, timeout=timeout, context=ctx)

It will load the certs correctly and be able to verify the certs.

Expected behavior

I expect this to be able to work with self signed certs if the cert chain is already added to the vm trust store. I don't think its reasonable to change default open ssl behavior on all vms just to accommodate this. The client should be able to look at the vm trust store and verify certificates being presented to it.

Current behavior

Already described above in the bug description.

To Reproduce

1. Deploy the pro-airgapped-server charm
2. Use haproxy to front this contracting server from step 1 and add a https endpoint with self-signed certificates
3. Deploy the ubuntu-advantage client charm
4. Configure the u-a client charm to talk to the https endpoint of the pro airgapped server

System information:

• Ubuntu release: 22.04.4 LTS
• Pro Client version: 32.3~22.04

Additional context

N/A

[orndorffgrant]

Thanks for the detailed report @nishant-dash!

I've been spending some time trying to reproduce this. I haven't completely nailed down what's happening for you but I want to report on my progress thus far.

I've been focusing on jammy's python3.10 urllib.request.urlopen behavior with respect to system certs.

Code analysis

First a quick code walk. Calling urlopen with just a url brings you here: https://github.com/python/cpython/blob/v3.10.13/Lib/urllib/request.py#L139-L140

If no ca* args are passed you end up down here: https://github.com/python/cpython/blob/v3.10.13/Lib/urllib/request.py#L213

build_opener() will include an HTTPSHandler https://github.com/python/cpython/blob/v3.10.13/Lib/urllib/request.py#L583-L584

HTTPSHandler's https_open will be called for an https connection, which uses http.client.HTTPSConnection https://github.com/python/cpython/blob/v3.10.13/Lib/urllib/request.py#L1390-L1392

HTTPSConnection will initiate a context if it isn't passed one using ssl_create_default_https_context: https://github.com/python/cpython/blob/main/Lib/http/client.py#L1463-L1464 https://github.com/python/cpython/blob/main/Lib/http/client.py#L804-L814

That will eventually call load_default_certs https://github.com/python/cpython/blob/main/Lib/ssl.py#L718-L722

Which eventually calls https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_default_verify_paths

And we can use this to verify it is set properly https://docs.python.org/3/library/ssl.html#ssl.get_default_verify_paths

In a jammy lxd container:

root@test-client:~# python3 -c "import ssl; print(ssl.get_default_verify_paths())"
DefaultVerifyPaths(cafile=None, capath='/usr/lib/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs')
root@test-client:~# ll /usr/lib/ssl
total 12
drwxr-xr-x 1 root root   54 Aug 23 02:11 ./
drwxr-xr-x 1 root root 1676 Aug 23 02:17 ../
lrwxrwxrwx 1 root root   14 Mar 16  2022 certs -> /etc/ssl/certs/
drwxr-xr-x 1 root root   36 Aug 23 02:11 misc/
lrwxrwxrwx 1 root root   20 Jul 30 15:18 openssl.cnf -> /etc/ssl/openssl.cnf
lrwxrwxrwx 1 root root   16 Mar 16  2022 private -> /etc/ssl/private/
root@test-client:~# 

That is interesting that the default configured cafile for openssl is a file that doesn't exist, but the capath exists and points to the expected location.

Testing

This whole stack has a bunch of abstractions and is pretty hard to follow so it's possible I missed something. So I decided to test it.

You should be able to mostly copy-paste these steps to reproduce. They create two lxd containers, configure one to be a server with a cert signed by a self-signed CA, and configure the other to trust it using the method documented here.

lxc launch ubuntu-daily:jammy test-client
lxc launch ubuntu-daily:jammy test-server
lxc shell test-server
# now in test-server container
openssl req -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -out ca.crt -keyout ca.key -subj "/C=CN/ST=BJ/O=STS/CN=CA"
openssl genrsa -out test-server.lxd.key
openssl req -new -key test-server.lxd.key -out test-server.lxd.csr -subj "/C=CN/ST=BJ/O=STS/CN=test-server.lxd"
cat > data.ext <<EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
subjectAltName = @alt_names
[alt_names]
DNS.1 = test-server.lxd
EOF
openssl x509 -req -in test-server.lxd.csr -out test-server.lxd.crt -sha256 -CA ca.crt -CAkey ca.key -CAcreateserial -days 3650 -extfile data.ext
apt update && apt install nginx
vim /etc/nginx/sites-enabled/default

Remove the server_name _; line and add the following in its place in the default site config

server {
...
       listen 443 ssl default_server;
       listen [::]:443 ssl default_server;
...
       server_name         test-server.lxd;
       ssl_certificate     /root/test-server.lxd.crt;
       ssl_certificate_key /root/test-server.lxd.key;
...
}

Then back to the shell in test-server

systemctl restart nginx
exit # now on the host
lxc file pull test-server/root/ca.crt .
lxc file push ./ca.crt test-client/usr/local/share/ca-certificates/ourselfsignedca.crt
lxd shell test-client # now in the test-client container
# first test failures before
curl https://test-server.lxd
openssl s_client -connect test-server.lxd:443
python3 -c "from urllib import request; print(request.urlopen('https://test-server.lxd').read().decode())"
# configure system to trust the ca
update-ca-certificates
# all should work now
curl https://test-server.lxd
openssl s_client -connect test-server.lxd:443
python3 -c "from urllib import request; print(request.urlopen('https://test-server.lxd').read().decode())"

That demonstrates that urlopen, when called without a particular context, does trust the system CAs.

Questions

So now the question is: what is different between this example and the error case you've described?

Something I noticed while putting this together is that curl seems generally happier to accept certs. If the client system is only configured to trust the test-server cert itself (as opposed to the ca cert), then curl is happy but openssl complains.

With that in mind: how was the self-signed cert that is generating errors created? what is it's relationship to the cert being used by the server?

The other interesting question is why does it work when you specify cafile to openssl? Is openssl treating cafile differently from a cert found in capath? I haven't dug any deeper into this yet.

[orndorffgrant]

Related: pro should probably have a feature to explicitly set a trusted cert separate from the system's certs.

[nishant-dash]

I should add that i dont get a failed to verify completely error with openssl/urllib but rather

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)>

[nishant-dash]

Although different, I can confirm Grant's reproducer works but the certificates I have fails with the self signed error 19 I mentioned previously

[nishant-dash]

this works

SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt  ubuntu-advantage status --all
SERVICE          AVAILABLE  DESCRIPTION
anbox-cloud      yes        Scalable Android in the cloud
cc-eal           no         Common Criteria EAL2 Provisioning Packages
esm-apps         yes        Expanded Security Maintenance for Applications
esm-infra        yes        Expanded Security Mai
...

[orndorffgrant]

To summarize, pro does trust the system certs, but openssl can be picky about the details of those certs.

If anyone is trying to trust a particular self-signed cert that openssl doesn't trust when merely added to the system trust store, SSL_CERT_FILE is a way to tell openssl to trust it explicitly.

I'll leave this bug open to represent the feature request for a config option in pro to explicitly trust a special cert that is not in the system trust store. There is precedent for this. For example, see the ca-certs configuration field for canonical-livepatch.

Editing the title and description to make this clearer.