search

Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
4.08k stars 549 forks source link

Let's Encrypt fails to renew or create certificate, no errors #6058

Open visualwritings opened 5 months ago

visualwritings commented 5 months ago

We ran into an issue with Let's Encrypt not renewing our certificate anymore. We're running the latest version of MeshCentral on an Ubunut 20.04LTS machine.

We restarted MC this morning, which resulted in the following output from the leevents command:

4/26/2024 9:55:34 AM - Getting certs from local store (Production) 4/26/2024 9:55:34 AM - Reading certificate files 4/26/2024 9:55:34 AM - Setting LE cert for default domain. 4/26/2024 9:55:34 AM - Setting LE cert for domain dvrsolutions. 4/26/2024 9:55:40 AM - Certificate has -1 day(s) left. 4/26/2024 9:55:40 AM - Asking for new certificate because of expire time. 4/26/2024 9:55:40 AM - Generating private key... 4/26/2024 9:55:41 AM - Setting up ACME client... 4/26/2024 9:55:41 AM - Creating certificate request... 4/26/2024 9:55:41 AM - Requesting certificate from Let's Encrypt... 4/26/2024 9:55:45 AM - Succesful response to challenge. 4/26/2024 9:55:46 AM - Succesful response to challenge. 4/26/2024 9:55:46 AM - Succesful response to challenge. 4/26/2024 9:55:47 AM - Succesful response to challenge. 4/26/2024 9:55:47 AM - Succesful response to challenge. 4/26/2024 9:55:47 AM - Succesful response to challenge.

To check if it might be a permission issue we removed the production.* files from the letsencrypt directory and tried again, with the same result.

We eventually got it sorted by setting skipchallengeverification to true in the config. However, it has been working pretty flawless since 2021, so it is a bit odd this suddenly occurs. No other errors were encounted. I checked the code for LetsEncrypt and found that none of the log-messages that usually should follow "Requesting certificate from Let's Encrypt..." were outputted, so it seems the process silently fails after "Succesful response to challenge".

From what I understand from other related issues here the skipchallengeverification is a self-check used by the letsencrypt module, but I'm entirely sure what the implications are of skipping this check.

In short, for now it is working again, but depending on the cause of the issue I figured you should be made aware of this. Besides that, as mentioned I do not fully understand the implications of skipchallengeverification.

PathfinderNetworks commented 5 months ago

This LetsEncrypt renewal issue is widespread this month due to changes LetsEncrypt has made- especially with the addition of more verification servers located in countries that hadn't been used previously.
By any chance do you use geolocation blocking at your firewall? I do and block most of the world- especially much of Eastern Europe and Asian countries that are hacking hotspots. If so, this is almost certainly the issue you are seeing. I've moved most of my services over to ZeroSSL as a result. And am hoping ZeroSSL support can be added to MeshCentral.

Here is the notice from LetsEncrypt about this: https://community.letsencrypt.org/t/unexpected-renewal-failures-during-april-2024-please-read-this/216830

visualwritings commented 5 months ago

@PathfinderNetworks I guess it could be related, but the output mentions the certificate was properly retrieved, but it just wasn't saving the cert files. It didn't give any error or notice, the renew / create process just quit without notice.

PathfinderNetworks commented 5 months ago

My apologies, you are correct. That is a completely different issue. The issue I mentioned results in the verification failing the step before a certificate is issued.