certgen fails to update public cert

[gainsley]

Certgen failed to update the public cert for the controller's access api endpoint.

This could be seen with the following command:

echo | openssl s_client -showcerts -connect eu.ctrl.{DOMAIN}:41001 2>/dev/null | openssl x509 -inform pem -noout -text

The fix was to go into the certgen pod and run the renew command manually. The renew command runs as part of cron:

# On global k8s:
kubectl exec -it certgen-644f57f6f6-7h7qd -- bash
## In pod:
/etc/letsencrypt/live# openssl x509 -in eu.ctrl.DOMAIN/cert.pem -text

more /etc/crontab
9 4,16 * * * root certbot renew >/proc/1/fd/1 2>/proc/1/fd/2

certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/_.abcdef.edgexr.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/_.dme.abc.edgexr.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for *.dme.abc.edgexr.org
Waiting 10 seconds for DNS changes to propagate

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/_.abc-01-abc.eu.app.abc.edgexr.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for *.abc-01-acb.eu.app.abc.edgexr.org
Waiting 10 seconds for DNS changes to propagate

Certbot failed to authenticate some domains (authenticator: dns-cloudflare). The Certificate Authority reported these problems:
  Domain: abc-01-abc.eu.app.abc.edgexr.org
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.abc-01-abc.eu.app.abc.edgexr.org - check that a DNS record exists for this domain

Hint: The Certificate Authority failed to verify the DNS TXT records created by --dns-cloudflare. Ensure the above domains are hosted by this DNS provider, or try increasing --dns-cloudflare-propagation-seconds (currently 10 seconds).

Failed to renew certificate _.abc-01-abc.eu.app.abc.edgexr.org with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/_.abc-gpu-abc.eu.app.abc.edgexr.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/eu.ctrl.abc.edgexr.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for eu.ctrl.abc.edgexr.org
Waiting 10 seconds for DNS changes to propagate

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following certificates are not due for renewal yet:
  /etc/letsencrypt/live/_.acb-abc.eu.app.abc.edgexr.org/fullchain.pem expires on 2023-02-19 (skipped)
  /etc/letsencrypt/live/_.abc-gpu-abc.eu.app.abc.edgexr.org/fullchain.pem expires on 2023-03-16 (skipped)
The following renewals succeeded:
  /etc/letsencrypt/live/_.dme.abc.edgexr.org/fullchain.pem (success)
  /etc/letsencrypt/live/eu.ctrl.abc.edgexr.org/fullchain.pem (success)

The following renewals failed:
  /etc/letsencrypt/live/_.abc-01-abc.eu.app.abc.edgexr.org/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

May need to increase 10sec timeout for validation.

[gainsley]

This was actually done for certgen cronttab, it's just the image was not updated.