Update image for crystallang/crystal:1.1.1 (because LE certificate expired) - consider autobuild of all images regularly?

[daliborfilus]

Hi, as you may know, Let's Encrypt Root certificate has expired. Because of that, ca-certificate package has to be updated.

I've run into an issue, where running docker build in my CI pipeline fails, because of that expired certificate.

dev:~# docker pull crystallang/crystal:1.1.1
1.1.1: Pulling from crystallang/crystal
Digest: sha256:ae8f53cc10b9cfdd786682cc420cce1097a06fc2e0db0ab4647965df64f25951
Status: Image is up to date for crystallang/crystal:1.1.1
docker.io/crystallang/crystal:1.1.1

dev:~# docker run --rm -ti crystallang/crystal:1.1.1 git clone https://my-le-domain.com/test/test.git
Cloning into 'test'...
fatal: unable to access 'https://my-le-domain.com/test/test.git/': server certificate verification failed. CAfile: none CRLfile: none

(If you do this via shards install, you just get:

$ shards install
Resolving dependencies
Fetching my-le-domain.com/test/test.git
Failed to clone my-le-domain.com/test/test.git

with no explanation why... I had to do manual git clone in CI config to find out the real cause.)

I have to run apt update && apt install ca-certificate for this to work. Interestingly, curl on the same domain works (don't know why). Updating ca-certificates package removes one CA certificate and updates none. Maybe it has to do something with that.

Anyway, the current image for current crystal/crystallang:1.1.1 is two months old now.

Are there any plans to update all (at least current, not EOL) versions of crystal images automatically?

Patches in upstream ubuntu:focal container maybe should be propagated downstream to your image... what do you think?

Base ubuntu:focal image doesn't include ca-certificates package, nor git-core package, so you install that yourself. Autobuilding your images could aleviate such problems in the future...

[straight-shoota]

There is no established mechanism for updating the docker images yet. So far we have just updated the images manually if that was necessary. I fully agree that an automated workflow would be very helpful, though.

However, as a related note: our plan is to try to get the images for stable releases into https://github.com/docker-library/official-images (#126). Then our images build from this repository would only be nightly and maintenance releases which have a short time span that probably doesn't need any updates.

---

On the original problem:

I'm actually surprised about this. crystal/crystallang:1.1.1 uses ubuntu:20.04 as base image. The default version of ca-certificates includes the new Let's Encrypt root certificate ISRG_Root_X1. And the openssl version (1.1.1f) is recent enough to stop at that trusted certificate and not try to validate the (now invalid) DST_Root_CA_X3 if it's also in the chain.

This setup should really work with Let's Encrypt domains. Thus it's no surprise that raw curl works. That's expected. But git should work as well. I can only assume that it somehow uses an older TLS implementation that still tries to verify X3. But it should be using the same curl internally. The only change brought by the ca-certificates update is removing the X3 certificate. That's fine, but it really shouldn't be necessary.

I tried connecting to a Let's Encrypt domain (with both X1 and X3 in the cert chain) via git clone and that works. So there might to be something special going on with your endpoint? Maybe you could get some more details with GIT_CURL_VERBOSE=1.

[daliborfilus]

Having #126 official images in docker hub as "crystal" will be very nice. Looking forward to that! That means that you'll be able to setup automated builds via Docker Hub and won't have to do it yourself, correct? So this issue can be closed. EDIT: Found out now that autobuilds doesn't mention scheduled building, so... we'll see.

On the problem:

ldd shows that the git binary doesn't link against the libcurl (nor other curl), so maybe it uses some curl source code directly (or uses it's static version)? That would explain different behavior vs. curl.

EDIT after closing: git package depends on libcurl3-gnutls, maybe it interacts with it via glibc dynamically.

Running git clone with GIT_CURL_VERBOSE=1 (thanks for that, I didn't know about this flag) shows the git binary doesn't mention

dev:~# docker run --rm -ti crystallang/crystal:1.1.1 bash
oot@2f2188a7d0b0:/# export GIT_CURL_VERBOSE=1
root@2f2188a7d0b0:/# git clone https://my-le-domain.com/test/test.git
Cloning into 'test'...
* Couldn't find host my-le-domain.com in the .netrc file; using defaults
*   Trying [ip-redacted]:443...
* TCP_NODELAY set
* Connected to my-le-domain.com ([ip-redacted]) port 443 (#0)
* found 387 certificates in /etc/ssl/certs
* ALPN, offering h2
* ALPN, offering http/1.1
* SSL connection using TLS1.3 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification failed. CAfile: none CRLfile: none
* Closing connection 0
fatal: unable to access 'https://my-le-domain.com/test/test.git/': server certificate verification failed. CAfile: none CRLfile: none

root@2f2188a7d0b0:/# curl -v https://my-le-domain.com/test/test.git
bash: curl: command not found
root@2f2188a7d0b0:/# apt update
[...]

# libcurl is not installed:
root@2f2188a7d0b0:/# apt install curl
[...]
The following additional packages will be installed:
  libcurl4
The following NEW packages will be installed:
  curl libcurl4
[...]
Preparing to unpack .../libcurl4_7.68.0-1ubuntu2.7_amd64.deb ...
Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.7) ...
Selecting previously unselected package curl.
Preparing to unpack .../curl_7.68.0-1ubuntu2.7_amd64.deb ...
Unpacking curl (7.68.0-1ubuntu2.7) ...
Setting up libcurl4:amd64 (7.68.0-1ubuntu2.7) ...
Setting up curl (7.68.0-1ubuntu2.7) ...

# libcurl not linked:
root@2f2188a7d0b0:/# ldd $(which git)
    linux-vdso.so.1 (0x00007ffef97f8000)
    libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f58294a4000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5829488000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5829465000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5829273000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f5829872000)

root@2f2188a7d0b0:/# git --version
git version 2.25.1

root@2f2188a7d0b0:/# curl https://my-le-domain.com/test/test.git -v
*   Trying [ip-redacted]:443...
* TCP_NODELAY set
* Connected to my-le-domain.com ([ip-redacted]) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=my-le-domain.com
*  start date: Oct  7 13:21:51 2021 GMT
*  expire date: Jan  5 13:21:50 2022 GMT
*  subjectAltName: host "my-le-domain.com" matched cert's "my-le-domain.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET /test/test.git HTTP/1.1
> Host: my-le-domain.com
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Thu, 07 Oct 2021 20:51:01 GMT
# redirects to login...

Maybe the libcurl version used by git reads the certificates in different order and tries to match something in the chain.

Anyway, thanks for your reply and your debugging help.