Closed MarkEWaite closed 1 year ago
Let's check again the Monday 10 of April: the certbot renew
command is run once a day, at 06:00am UTC, and is expected to renew certificates 1 month before expiration
Worth another check Tuesday 11 April 2023. since that will be 29 days before the expiration of the updates.jenkins.io certificate.
As of 22:40 UTC 10 April 2023 the certificate is not renewed.
Operations done with the help of @smerle33 earlier today, on the VM pkg.origin.jenkins.io
(which hosts the 2 updates.jenkins* services):
/etc/letsencrypt/live
, as symlinks. This is expected.certbot renew -q
) as expected (per https://github.com/jenkins-infra/jenkins-infra/blob/b92ba6e7d1e99c9c157af1a2a77a12514be45072/hieradata/common.yaml#L89-L91)certbot renew --dry-run
command to ensure that the toolchain is present and valid (certbot
CLI, the certbot plugins including apache
and dns-azure
, the expected python
installation, etc.) => the dry run reported success (e.g. all certificates were renewed but not persisted using the Letsencrypt staging area).
--dry-run
flag does NOT check if a certificate should be renewed (1 month before epxiratiojn) or not: it runs on all certificates wether or not they have to be renewedcertbot
runs in quiet mode, we are blind.
/var/log/letsencrypt/letsencrypt.log
was already overridin by our dry run testcertbot renew
command manually on the VM:
pkg.origin.jenkins*
weren't renewed since they already were the 17th March 2023. It's expected (not in the "1 month before expiration" time window) ✅ updates.jenkins.io
and updates.jenkins-ci.org
certificates were renewed with the same new expiration date: 10th July 2023 ✅ Next steps before closing this issue:
-q
flag and would keep history of the renwal logs, making us less blind when a renew fails~event in place for the 12th of june (monday)
Update on the "Puppet" specifics:
letsencrypt::certonly
classe, while we use letsencrypt:renew
. Can't be usedletsencrypt::renew_additional_args
but it only appends to the crontab command-q
flags is mandatoryHowever, I was able to check the following:
grep CRON /var/log/syslog*
shows the log lines related to crontab executions. Grepping on certbot
, I was able to check that it ran today (11 of April) but not the 10th.certbot
as per https://github.com/voxpupuli/puppet-letsencrypt/issues/164 which collides with the normal renewal => cleaned it upcp -r /etc/letsencrypt /root/bkp-letsencrypt-20230411
apt-get remove --purge certbot
cp -r /root/bkp-letsencrypt-20230411 /etc/letsencrypt
certbot renew
systemctl reload apache2
systemctl restart apache2
Reopening as the certificate has not renewed, again.
With @smerle33 we diagnosed the following elements:
certbot renew -q
run everyday at 06:00am UTC (defined in https://github.com/jenkins-infra/jenkins-infra/blob/14372daa6923c86b920539a12872022ca19bb82e/hieradata/common.yaml#L90-L92).syslog
) but no certificate renewed.certbot renew --dry-run
manually shows that certificate should be renewed as expectedSo we hacked a bit the crontab on the pkg
VM:
root
crontab with crontab -e
to write the output to a log file, get the stderr along stdout outputs, remove the quiet and run it on a time that fit our analysis (21 16 * * * crontab renew >/var/log/certbot-debug.log 2>&1
)=> it surfaced in the following error error: unknown command "renew", see 'snap help'.
=> trying a 28 16 * * * which certbot >/var/log/certbot-debug.log 2>&1
showed that /usr/bin/certbot
is used.
We relaized that, on some machines, the /usr/bin/certbot
file exists and is a symlink to /usr/bin/snap
explaining the error. Most probably a leftover from my tentatives to use snap package for certbot :'(
The symlink was removed from the following machines yesterday, and we'll wait today to see if the certificates are now renewed:
Checking the renewal today on pkg VM: no renewal.
After another crontab hacking, 34 10 * * * certbot plugins --text > /var/log/certbot-renew.log 2>&1
, the following error was surfaced: /bin/sh: 1: certbot: not found
=> we should check that /usr/local/bin
is part of the crontab's PATH
=> and/or we should stop rely on the Puppet module and manage the crontab item ourselve to benefit from:
New crontab applied, let's wait for tomorrow to check for certificates renewal
Confirmed that the certificates were renewed successfully and automatically earlier today:
➜ SERVER_NAME=updates.jenkins.io; PORT=443; echo -n Q | openssl s_client -servername {SERVER_NAME} -connect {SERVER_NAME}:{PORT} | openssl x509 -noout -dates
getaddrinfo: nodename nor servname provided, or not known
connect:errno=22
unable to load certificate
8351849984:error:09FFF06C:PEM routines:CRYPTO_internal:no start line:/AppleInternal/Library/BuildRoots/ff32e6fb-db00-11ed-a068-428477786501/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/crypto/pem/pem_lib.c:694:Expecting: TRUSTED CERTIFICATE
➜ SERVER_NAME=pkg.origin.jenkins.io; echo -n Q | openssl s_client -servername ${SERVER_NAME} -connect ${SERVER_NAME}:443 | openssl x509 -noout -dates
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = pkg.origin.jenkins.io
verify return:1
DONE
notBefore=Jun 22 05:05:18 2023 GMT
notAfter=Sep 20 05:05:17 2023 GMT
➜ SERVER_NAME=updates.jenkins-ci.org; echo -n Q | openssl s_client -servername ${SERVER_NAME} -connect ${SERVER_NAME}:443 | openssl x509 -noout -dates
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = updates.jenkins-ci.org
verify return:1
DONE
notBefore=Jun 22 05:05:40 2023 GMT
notAfter=Sep 20 05:05:39 2023 GMT
Please note that ci.jenkins.io certificates were also renewed (same reasons, same blockage, same fix):
➜ SERVER_NAME=ci.jenkins.io; echo -n Q | openssl s_client -servername ${SERVER_NAME} -connect ${SERVER_NAME}:443 | openssl x509 -noout -dates
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = ci.jenkins.io
verify return:1
DONE
notBefore=Jun 22 05:00:45 2023 GMT
notAfter=Sep 20 05:00:44 2023 GMT
Service(s)
Update center
Summary
The Jenkins update center is available from two different URLs:
The SSL certificate on the first URL https://updates.jenkins.io expires May 9, 2023. That is 31 days from now. If we renew it within the next week, that will be a comfortable margin for renewal
The SSL certificate on the second URL https://updates.jenkins-ci.org expires May 1, 2023. That is 23 days from now. If we renew it within the next week, that will be a comfortable margin for renewal
Since they are the same machine, it is a little surprising that the SSL certificates expire at different times
Reproduction steps