Removing `DST Root CA X3` certificate from the docker image

[alibo]

What happened:

The docker image of openshift-acme uses openssl 1.0.2 and ca-certificates is not updated:

docker run -it --rm --entrypoint="/bin/bash" docker.io/tnozicka/openshift-acme:v0.8.0
$ rpm -qa | grep openssl
openssl-libs-1.0.2k-16.el7_6.1.x86_64
$ trust list | grep DST
    label: DST Root CA X3
$ trust list | grep ISRG
    label: ISRG Root X1

What you expected to happen:

After today's Let's encrypt changes, it cannot connect to the let's encrypt servers unless the TLS verification is disabled or DST Root CA X3 is removed.

How to reproduce it (as minimally and precisely as possible):

# you need to install wget and faketime packages first (faketime doesn't work with statically linked binaries)
faketime '1 Oct 2021' wget https://acme-staging-v02.api.letsencrypt.org/directory -O-

Anything else we need to know?:

it can be easily fixed by updating ca-certificates pkg: https://blog.devgenius.io/rhel-centos-7-fix-for-lets-encrypt-change-8af2de587fe4

@tnozicka

[alibo]

In the v0.9 image in quay.io, the expiring certificate still exists, but its OpenSSL is updated to 1.1.1c:

docker run -it --rm --entrypoint="/bin/bash" quay.io/tnozicka/openshift-acme:controller-0.9
$ trust list | grep DST
    label: DST Root CA X3
$ trust list | grep ISRG
    label: ISRG Root X1
$ rpm -qa | grep openssl
openssl-libs-1.1.1c-16.el8_2.x86_64

[hufon]

I try to update my own image with latest ubi8 and we still have the problem. How does the controller builds the certificate chain ?

[hufon]

i think it's not just a matter to remove the certificate from the docker image, but have a way to pass an alternativeChain to the acme client...

[tnozicka]

I could cut a new version with openssl but I am not sure if it helps. I haven't look deep, but isn't that more of a client issue? The controller also talks to the https acme directory service but the validation is done by golang and I don't think it delegates to openssl.

How does the issue manifest for the controller? Given your example with faketime, the real time is now even further and I can curl the ACME directory service just fine from the image:

$ podman run -it --rm --entrypoint=/bin/bash docker.io/tnozicka/openshift-acme:latest -c 'curl -LI https://acme-staging-v02.api.letsencrypt.org/directory'
HTTP/1.1 200 OK

[alibo]

@tnozicka I checked it again with wget and https://letsencrypt.org/(seems it uses defaultChain unlike acme-staging-v02.api.letsencrypt.org). It still has issue:

run -it --rm --entrypoint="/bin/bash" docker.io/tnozicka/openshift-acme:latest
wget https://letsencrypt.org/
--2021-10-05 07:39:55--  https://letsencrypt.org/
Resolving letsencrypt.org (letsencrypt.org)... 18.159.128.50, 161.35.218.92, 2a03:b0c0:3:d0::143f:a001, ...
Connecting to letsencrypt.org (letsencrypt.org)|18.159.128.50|:443... connected.
ERROR: cannot verify letsencrypt.org's certificate, issued by '/C=US/O=Let\'s Encrypt/CN=R3':
  Issued certificate has expired.
To connect to letsencrypt.org insecurely, use `--no-check-certificate'.

but curl is ok: (maybe it uses a different OpenSSL library, I don't know! As you said, it can also be true for golang apps as well, especially the static binaries)

curl -LIsq https://letsencrypt.org/ | head -n1
HTTP/1.1 200 OK

Anyway, we've updated the image to quay.io/tnozicka/openshift-acme:controller-0.9 before let's encrypt changes and we haven't noticed any issues since then, but I'm not sure about docker.io/tnozicka/openshift-acme:v0.8.0 and docker.io/tnozicka/openshift-acme:latest. If you didn't see any issues, maybe we can close this then?

---

@hufon this issue is just for server-to-server communications (in this case, openshift-acme <---> let's encrypt).

[tnozicka]

yeah, the controller doesn't link openssl, so I guess there is no point in bumping that for the resolution.