MariaDB / mariadb-docker

Docker Official Image packaging for MariaDB
https://mariadb.org
GNU General Public License v2.0
770 stars 438 forks source link

TLS certificate for MariaDB archive broken; can't access apt archive #394

Closed peterthomassen closed 3 years ago

peterthomassen commented 3 years ago

When running apt-get update within a mariadb:10.3 container, the following error occurs:

# apt-get update
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Ign:3 https://archive.mariadb.org/mariadb-10.3.31/repo/ubuntu focal InRelease
Err:6 https://archive.mariadb.org/mariadb-10.3.31/repo/ubuntu focal Release
  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 138.201.152.105 443]
Get:7 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [794 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [580 kB]
Get:10 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [30.1 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1133 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [630 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [33.3 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1580 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1082 kB]
Get:19 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [6310 B]
Get:20 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [2668 B]
Reading package lists...
E: The repository 'http://archive.mariadb.org/mariadb-10.3.31/repo/ubuntu focal Release' does not have a Release file.

As we run this command to install extra packages into our image, this breaks our build chain (and I'm sure it breaks lots of other people's build chains).

I am not sure if this is an issue within the MariaDB image or with the archive server; I decided to post it here. (Hint: The issue is possibly related to today's expiration of Let's Encrypt's DST Root CA X3 certificate. Perhaps the image's trust store is outdated?)

peterthomassen commented 3 years ago

Verified that this also occurs with mariadb:latest.

grooverdan commented 3 years ago

So looking at https://www.ssllabs.com/ssltest/analyze.html?d=archive.mariadb.org its only once you expand the certificate paths that you see the "In trust store DST Root CA X3" expired.

As this is an intrust store certificate expired, the package in the container needs to be updated.

$ podman run --rm -ti mariadb:latest bash
root@3578c2b04fba:/# apt-get update
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]                                                     
Get:3 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]                                                
Ign:1 https://archive.mariadb.org/mariadb-10.6.4/repo/ubuntu focal InRelease                                             
Get:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]                                                  
Err:4 https://archive.mariadb.org/mariadb-10.6.4/repo/ubuntu focal Release                                               
  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 138.201.152.105 443]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]                                                
Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [794 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]                                                                                                                                       
Get:11 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]                                                                                                                                        
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [630 kB]                                                                                                                                
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [33.3 kB]                                                                                                                               
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1580 kB]                                                                                                                                     
Get:15 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1133 kB]                                                                                                                                   
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1082 kB]                                                                                                                                 
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [6310 B]                                                                                                                                
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [2668 B]                                                                                                                                    
Get:19 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [580 kB]                                                                                                                              
Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [30.1 kB]                                                                                                                             
Reading package lists... Done                                                                                                                                                                                           
E: The repository 'http://archive.mariadb.org/mariadb-10.6.4/repo/ubuntu focal Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
root@3578c2b04fba:/# apt-get -s upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  ca-certificates libgcrypt20 libmysqlclient21 libprocps8 libsystemd0 libudev1 procps
7 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Inst libsystemd0 [245.4-4ubuntu3.11] (245.4-4ubuntu3.13 Ubuntu:20.04/focal-updates [amd64])
Conf libsystemd0 (245.4-4ubuntu3.13 Ubuntu:20.04/focal-updates [amd64])
Inst libgcrypt20 [1.8.5-5ubuntu1] (1.8.5-5ubuntu1.1 Ubuntu:20.04/focal-updates, Ubuntu:20.04/focal-security [amd64])
Conf libgcrypt20 (1.8.5-5ubuntu1.1 Ubuntu:20.04/focal-updates, Ubuntu:20.04/focal-security [amd64])
Inst libudev1 [245.4-4ubuntu3.11] (245.4-4ubuntu3.13 Ubuntu:20.04/focal-updates [amd64])
Conf libudev1 (245.4-4ubuntu3.13 Ubuntu:20.04/focal-updates [amd64])
Inst libprocps8 [2:3.3.16-1ubuntu2.2] (2:3.3.16-1ubuntu2.3 Ubuntu:20.04/focal-updates [amd64])
Inst procps [2:3.3.16-1ubuntu2.2] (2:3.3.16-1ubuntu2.3 Ubuntu:20.04/focal-updates [amd64])
Inst ca-certificates [20210119~20.04.1] (20210119~20.04.2 Ubuntu:20.04/focal-updates, Ubuntu:20.04/focal-security [all])
Inst libmysqlclient21 [8.0.26-0ubuntu0.20.04.2] (8.0.26-0ubuntu0.20.04.3 Ubuntu:20.04/focal-updates [amd64])
Conf libprocps8 (2:3.3.16-1ubuntu2.3 Ubuntu:20.04/focal-updates [amd64])
Conf procps (2:3.3.16-1ubuntu2.3 Ubuntu:20.04/focal-updates [amd64])
Conf ca-certificates (20210119~20.04.2 Ubuntu:20.04/focal-updates, Ubuntu:20.04/focal-security [all])
Conf libmysqlclient21 (8.0.26-0ubuntu0.20.04.3 Ubuntu:20.04/focal-updates [amd64])

root@3578c2b04fba:/# apt-get upgrade -y 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  ca-certificates libgcrypt20 libmysqlclient21 libprocps8 libsystemd0 libudev1 procps
7 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2406 kB of archives.
After this operation, 1024 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsystemd0 amd64 245.4-4ubuntu3.13 [270 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libgcrypt20 amd64 1.8.5-5ubuntu1.1 [420 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libudev1 amd64 245.4-4ubuntu3.13 [77.6 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libprocps8 amd64 2:3.3.16-1ubuntu2.3 [33.0 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 procps amd64 2:3.3.16-1ubuntu2.3 [233 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ca-certificates all 20210119~20.04.2 [145 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmysqlclient21 amd64 8.0.26-0ubuntu0.20.04.3 [1227 kB]
Fetched 2406 kB in 3s (728 kB/s)          
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 10093 files and directories currently installed.)
Preparing to unpack .../libsystemd0_245.4-4ubuntu3.13_amd64.deb ...
Unpacking libsystemd0:amd64 (245.4-4ubuntu3.13) over (245.4-4ubuntu3.11) ...
Setting up libsystemd0:amd64 (245.4-4ubuntu3.13) ...
(Reading database ... 10093 files and directories currently installed.)
Preparing to unpack .../libgcrypt20_1.8.5-5ubuntu1.1_amd64.deb ...
Unpacking libgcrypt20:amd64 (1.8.5-5ubuntu1.1) over (1.8.5-5ubuntu1) ...
Setting up libgcrypt20:amd64 (1.8.5-5ubuntu1.1) ...
(Reading database ... 10093 files and directories currently installed.)
Preparing to unpack .../libudev1_245.4-4ubuntu3.13_amd64.deb ...
Unpacking libudev1:amd64 (245.4-4ubuntu3.13) over (245.4-4ubuntu3.11) ...
Setting up libudev1:amd64 (245.4-4ubuntu3.13) ...
(Reading database ... 10093 files and directories currently installed.)
Preparing to unpack .../libprocps8_2%3a3.3.16-1ubuntu2.3_amd64.deb ...
Unpacking libprocps8:amd64 (2:3.3.16-1ubuntu2.3) over (2:3.3.16-1ubuntu2.2) ...
Preparing to unpack .../procps_2%3a3.3.16-1ubuntu2.3_amd64.deb ...
Unpacking procps (2:3.3.16-1ubuntu2.3) over (2:3.3.16-1ubuntu2.2) ...
Preparing to unpack .../ca-certificates_20210119~20.04.2_all.deb ...
Unpacking ca-certificates (20210119~20.04.2) over (20210119~20.04.1) ...
Preparing to unpack .../libmysqlclient21_8.0.26-0ubuntu0.20.04.3_amd64.deb ...
Unpacking libmysqlclient21:amd64 (8.0.26-0ubuntu0.20.04.3) over (8.0.26-0ubuntu0.20.04.2) ...
Setting up libmysqlclient21:amd64 (8.0.26-0ubuntu0.20.04.3) ...
Setting up ca-certificates (20210119~20.04.2) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
Updating certificates in /etc/ssl/certs...

0 added, 1 removed; done.
Setting up libprocps8:amd64 (2:3.3.16-1ubuntu2.3) ...
Setting up procps (2:3.3.16-1ubuntu2.3) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Processing triggers for ca-certificates (20210119~20.04.2) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
root@3578c2b04fba:/# apt-get update
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease                                                                        
Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease           
Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease                     
Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease                   
Get:1 https://archive.mariadb.org/mariadb-10.6.4/repo/ubuntu focal InRelease [7758 B]
Get:6 https://archive.mariadb.org/mariadb-10.6.4/repo/ubuntu focal/main amd64 Packages [17.3 kB]
Fetched 25.1 kB in 4s (5870 B/s)     
Reading package lists... Done

Which shows Ubuntu has issued the ca-certificates package update, and that ignoring the error, updating ca-certificates and the apt-get update again will work.

I think after https://github.com/docker-library/official-images/pull/10990 is merged, the dependent containers including mariadb will be updated like happened in https://github.com/docker-library/repo-info/commit/93bbed8119abf1d4171b46f59d2ee015fe44eb55.

Given this usually doesn't take too long, and I've no way to short cut this, lets leave it for a day or 2 to see how it resolves.

grooverdan commented 3 years ago

rebuild job for amd64: https://doi-janky.infosiftr.net/job/multiarch/job/amd64/job/mariadb/215/console

grooverdan commented 3 years ago

Magic of patience and fortuitous coincidence (Ubuntu updates where unrelated to this issue) paid off :smile_cat:

$ podman pull mariadb:10.6
Trying to pull docker.io/library/mariadb:10.6...
Getting image source signatures
Copying blob 8ff899ddf1cc done  
Copying blob 736a10760fd6 done  
Copying blob 6d6f45e0fb03 done  
Copying blob f3ef4ff62e0d done  
Copying blob a3e10ba5def7 done  
Copying blob 3a0645d99211 done  
Copying blob 373c21a041de done  
Copying blob ef565721758b done  
Copying blob 3d1a9ce482cc done  
Copying blob ecedf97e512c done  
Copying config b7220a722c done  
Writing manifest to image destination
Storing signatures
b7220a722ce2a763177738b8cb0b42b3602368100ee7cc7088fb0bcc96fea1c3

~ 
$ podman run -ti mariadb:10.6  apt-get update
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]                
Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1133 kB]                                     
Get:3 https://archive.mariadb.org/mariadb-10.6.4/repo/ubuntu focal InRelease [7758 B]  
Get:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]                  
Get:6 https://archive.mariadb.org/mariadb-10.6.4/repo/ubuntu focal/main amd64 Packages [17.3 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]                
Get:8 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [30.1 kB]            
Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [580 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB] 
Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [794 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]         
Get:13 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [33.3 kB]                                                
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1082 kB]                                                  
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [630 kB]                                                 
Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1580 kB]                                                      
Get:19 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [2668 B]                                                     
Get:20 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [6310 B]                                                 
Fetched 19.3 MB in 8s (2478 kB/s)                                                                                                        
Reading package lists... Done