Fresh certificates should be presented by nginx

[bickelj]

Following a successful renewal of certificates by the renewCerts.sh script running under the root crontab, an old certificate still appears to be presented by nginx.

./testssl.sh api.philanthropydatacommons.org
...
Start 2023-08-07 11:42:18
...
 Certificate Validity (UTC)   expires < 15 days (12) (2023-05-21 18:00 --> 2023-08-19 18:00)

On the machine:

# ls -la /etc/letsencrypt/archive/api.philanthropydatacommons.org/
total 68
drwxr-xr-x 2 root reverse-proxy 4096 Jul 20 20:57 .
drwxr-x--- 4 root reverse-proxy 4096 Mar 22 19:56 ..
-rw-r-xr-- 1 root reverse-proxy 1887 Mar 22 19:56 cert1.pem
-rw-r-xr-- 1 root reverse-proxy 1887 May 21 19:00 cert2.pem
-rw-r-xr-- 1 root reverse-proxy 1814 Jul 20 20:57 cert3.pem
-rw-r-xr-- 1 root reverse-proxy 3749 Mar 22 19:56 chain1.pem
-rw-r-xr-- 1 root reverse-proxy 3749 May 21 19:00 chain2.pem
-rw-r-xr-- 1 root reverse-proxy 3749 Jul 20 20:57 chain3.pem
-rw-r-xr-- 1 root reverse-proxy 5636 Mar 22 19:56 fullchain1.pem
-rw-r-xr-- 1 root reverse-proxy 5636 May 21 19:00 fullchain2.pem
-rw-r-xr-- 1 root reverse-proxy 5563 Jul 20 20:57 fullchain3.pem
-rw-r-x--- 1 root reverse-proxy 1704 Mar 22 19:56 privkey1.pem
-rw-r-x--- 1 root reverse-proxy 1704 May 21 19:00 privkey2.pem
-rw-r-x--- 1 root reverse-proxy 1704 Jul 20 20:57 privkey3.pem
# openssl x509 -text -in /etc/letsencrypt/archive/api.philanthropydatacommons.org/fullchain3.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:48:02:61:05:1c:11:0a:a2:9f:42:98:60:5c:50:59:8b:f0
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Jul 20 19:57:18 2023 GMT
            Not After : Oct 18 19:57:17 2023 GMT
        Subject: CN = api.philanthropydatacommons.org
...
# ls -la /etc/letsencrypt/live/api.philanthropydatacommons.org/
total 16
drwxr-xr-x 2 root reverse-proxy 4096 Jul 20 20:57 .
drwxr-x--- 4 root reverse-proxy 4096 Mar 22 19:56 ..
-rw-r-xr-- 1 root reverse-proxy  692 Mar 22 19:56 README
lrwxrwxrwx 1 root reverse-proxy   55 Jul 20 20:57 cert.pem -> ../../archive/api.philanthropydatacommons.org/cert3.pem
lrwxrwxrwx 1 root reverse-proxy   56 Jul 20 20:57 chain.pem -> ../../archive/api.philanthropydatacommons.org/chain3.pem
lrwxrwxrwx 1 root reverse-proxy   60 Jul 20 20:57 fullchain.pem -> ../../archive/api.philanthropydatacommons.org/fullchain3.pem
lrwxrwxrwx 1 root reverse-proxy   58 Jul 20 20:57 privkey.pem -> ../../archive/api.philanthropydatacommons.org/privkey3.pem

The script that runs weekly via cron, renewCerts.sh, includes this line which was expected to cause nginx to see the new certificates:

docker exec deploy_reverse-proxy_1 nginx -s reload \
    || fin "Failed to send reload signal to reverse proxy running in a container"

[bickelj]

To address the immediate issue of an invalid certificate this weekend, I ran sudo docker restart deploy_reverse-proxy_1 in test and then in prod, and now the latest certificates are presented. This does not discover or fix the underlying issue, however.

[bickelj]

Happened again where the reload was not enough. A similar targeted mitigation, restart only the reverse-proxy (nginx) container, this worked a few moments ago: docker-compose -f $(cat compose_current_file_name) restart reverse-proxy