Trouble debugging renew issue

[peturdainn]

Discovered that my server failed to renew its certificates, the error message mentioned unable to contact ACME server.

Needed to increase mod_md logging to get a meaningful error but still not clear what is going on, so details could be improved:

log sample of interest: `[Thu Jun 30 11:09:35.403461 2022] [md:debug] [pid 18561:tid 140264260130560] md_acme.c(776): get directory from https://acme-v02.api.letsencrypt.org/directory

[Thu Jun 30 11:09:35.421524 2022] [md:debug] [pid 18561:tid 140264260130560] md_curl.c(402): (20014)Internal error (specific information not available): request failed(77): Problem with the SSL CA cert (path? access rights?)

[Thu Jun 30 11:09:35.423197 2022] [md:warn] [pid 18561:tid 140264260130560] (20014)Internal error (specific information not available): md[myserver.be] while[Contacting ACME server for myserver.be at https://acme-v02.api.letsencrypt.org/directory] detail[Unsuccessful in contacting ACME server at https://acme-v02.api.letsencrypt.org/directory. If this problem persists, please check your network connectivity from your Apache server to the ACME server. Also, older servers might have trouble verifying the certificates of the ACME server. You can check if you are able to contact it manually via the curl command. Sometimes, the ACME server might be down for maintenance, so failing to contact it is not an immediate problem. Apache will continue retrying this.]

[Thu Jun 30 11:09:35.424752 2022] [md:error] [pid 18561:tid 140264260130560] (20014)Internal error (specific information not available): md[myserver.be] while[Contacting ACME server for myserver.be at https://acme-v02.api.letsencrypt.org/directory] detail[Unsuccessful in contacting ACME server at https://acme-v02.api.letsencrypt.org/directory. If this problem persists, please check your network connectivity from your Apache server to the ACME server. Also, older servers might have trouble verifying the certificates of the ACME server. You can check if you are able to contact it manually via the curl command. Sometimes, the ACME server might be down for maintenance, so failing to contact it is not an immediate problem. Apache will continue retrying this.]

[Thu Jun 30 11:09:35.426270 2022] [md:debug] [pid 18561:tid 140264260130560] md_result.c(254): (20014)Internal error (specific information not available): md[myserver.be] while[Contacting ACME server for myserver.be at https://acme-v02.api.letsencrypt.org/directory] detail[Unsuccessful in contacting ACME server at https://acme-v02.api.letsencrypt.org/directory. If this problem persists, please check your network connectivity from your Apache server to the ACME server. Also, older servers might have trouble verifying the certificates of the ACME server. You can check if you are able to contact it manually via the curl command. Sometimes, the ACME server might be down for maintenance, so failing to contact it is not an immediate problem. Apache will continue retrying this.]

[Thu Jun 30 11:09:35.427874 2022] [md:debug] [pid 18561:tid 140264260130560] md_reg.c(1158): (20014)Internal error (specific information not available): myserver.be: staging done `

(substituted my domain name with myserver.be)

I can SSH into the server and curl the ACME server without problem There's a /etc/ssl/ca-bundle.crt with read access available.

So I'm missing details on what the exact problem is

[icing]

It looks as if libcurl is unable to access the system CA certificates. The message part Problem with the SSL CA cert (path? access rights?) comes from libcurl itself.

What is the OS you are running on? Does the server user (often www-data have the proper permissions? Is it maybe chrooted? There are distributions like SELinux that try to restrict access to some processes..

[peturdainn]

This is on my QNAP NAS, QTS is debian-based, this has worked fine until a few months ago, problem is where does libcurl look for the certs? I'm not blaming mod_md, to be 100% clear, but I'm missing info to figure out why things broke

[peturdainn]

I have recreated the ca-bundle.crt, same location, same owner, same access rights. And now it works fine.

Go figure...

[icing]

Mysteries of the universe. Thanks for letting me know.