Closed icing closed 21 hours ago
Maybe this needs to be added to the README, but let's clarify here first.
The motivation for this discussion seems to be: "What happens when Lets Encrypt stops working (for whatever reason)? How can I protect my site?"
For any virtual https
host (with SSLEnging on
), Apache needs a certificate. For this reason, mod_md
will on such occasion create a self-signed certificate with all domain names of that MD. This is called a "fallback cert" by mod_md.
While the module is negotiating with Lets Encrypt for a "real" certificate, visitors to this site will see the fallback. Their browser will not trust it, but they may "click through". Once the ACME certificate is there, and the server is reloaded, the fallback is archived and replaced with the new cert.
Should the ACME server fail to give a new certificate, errors will be logged, notifications will be triggered (see MDMessageCmd
) and the fallback cert stays in place.
Let's assume the initial setup worked and you got certificates from Lets Encrypt. Then, for some reason, renewals stop working. What will mod_md
do?
First, it will continue to use the certificate it has. Even if it already expired. The client browser then will not trust it, but the user may "click through". The module will keep on trying to get you a new cert, using a backoff delay in retries, but do try that at least once per day or more often. It will tell you about this via MDMessageCmd
which is highly recommended.
With the default settings, a Lets Encrypt cert lives for 90 days and Apache will renew it 30 days before its lifetime runs out. So, this failure would need to go unfixed for 30 days before your users notice.
You can lower some risks of failures by adding a second ACME server to you config. See how to setup a failover CA in the documentation.
There is no way to configure mod_md
with your own fallback certificate. As mentioned above, when no certificate is there at all, it will create a self-signed fallback for you. If a cert exists, it will continue to use that, even when expired.
This seems as good as providing your own self-signed fallback, as in all cases users will get a warning and need to "click through". Now, if you are able to generate a valid fallback certificate at all times, you do not need mod_md in the first place. Then just use your own cert all the time.
"Click-through" does not happen in all cases, unfortunately.
A warning is not issued if a site has HSTS records present. You simply universally get a "cannot establish a secure connection" error, and not all browsers even indicate it is related to HSTS.
Thus, a cert (wildcard or otherwise), signed by our own "internal" CA which would allow us to continue to use things, until LE did its thing.
It would be useful to have the behavior of how the fallback cert is generated (perhaps just copied from somewhere else on disk?) configurable.
Second, note my logs entry above, it implies the existence of a fallback entry that is per domain:
configured from /usr/local/etc/apache24/md/domains/drivingdemocrats.org/fallback-cert.pem and /usr/local/etc/apache24/md/domains/drivingdemocrats.org/fallback-privkey.pem
Whereas the File Storage section of the readme only references those being in the main folder, and not the per-domain folders.
In the readme, you say "What is happening? At start up, mod_md generated a self-signed certificate for the new https host to use and switch that host to 503 responses. This made sure that your server started without delay and that your other hosts could start working." but there's no mention made that this is the "Fallback" cert mentioned.
===
And, third, finally, on the main apache site -- NO mention is made of how and when these fallback certs are generated. Maybe that needs a fix with an actual apache bug, but it would be nice to point them to a corrected readme here (i.e. mention the filenames in the readme, and in the File Storage section above.
This would not only affect the fallback on a new site, but especially the behaviour when a cert has expired, correct? This would only be possible on a server reload, however, since once a certificate has been loaded, it is fixed for the runtime of the server. Hmm...
As for the documentation, this can always be improved and you are welcome to add a PR for better explaining things. So far, the fallback cert was treated as an initial thing with no peculiar interest to users, since it usually disappears after some minutes.
However, your use case is new. To get the benefits you describe, you need clients with an updated root store, which is outside enterprises generally frowned upon. (If I understand your case correctly.)
I do not doubt that your motivations are valid. However, I'd like to spend my free time more on things benefitting a wider range of users.
Just an idea: if you have your internal "CA" anyway, why not give it an ACME server (pebble is quite uncomplicated) and configure that server as fallback for mod_md? That should give you exactly what you want.
That's a thought -- right now, my initial goal, as well as the one in the email from a couple years ago on apache users is "understand in which cases the fallback cert is used".
Ignoring HSTS (and just so I can suggest better docs), is there a way to configure mod_md to treat a site using the fallback cert as a valid site, instead of the 503?
Also, is there a global fallback cert, as mentioned in the file storage of the README, or is that a typo?
Ignoring HSTS (and just so I can suggest better docs), is there a way to configure mod_md to treat a site using the fallback cert as a valid site, instead of the 503?
No. I think your idea of what a "fallback" certificate is and what mod_md
calls its initial self-signed are different. We need to untangle that for a meaningful discussion.
Also, is there a global fallback cert, as mentioned in the file storage of the README, or is that a typo?
No, there is not for some considerable time. The documentation was not updated. I just removed the mention. Thanks for pointing out.
Okay, so:
Is that a correct understanding?
If so, I guess my only other question is (for documenting this), what params are used to generate that cert (key strength, algorithm, lifetime, subject name?)
My assumption of "fallback" (based purely on that word -- I would have used "initial" or "preliminary" or something) could have been any of:
But I see this isn't correct. Hopefully you can understand my confusion, and why I was mystified that it wasn't documented.
(To be clear, I'm not asking for it to be any of those, just explaining my thought process.)
Okay, so:
- The "fallback" (as named by the code) cert really is generated by apache on the first invocation of mod_md for a given domain, and it exists just so apache can start up, since it can't start up with a null cert.
- Even if you click through it, you get a 503, and that's going to happen until a valid cert is in place.
- There's no way to tell the code to use anything else on that first invocation
- and it will be used until an apachectl graceful happens -- that's the only way to advance the "State machine".
Is that a correct understanding?
Yes.
If so, I guess my only other question is (for documenting this), what params are used to generate that cert (key strength, algorithm, lifetime, subject name?)
It's all in the code. The generation is in md_crypt.c:1955
. Keys are generated according to the MDPrivateKeys
specified for the domain.
My assumption of "fallback" (based purely on that word -- I would have used "initial" or "preliminary" or something) could have been any of:
- A commercial cert you had bought and wanted to use prior to swinging over to full acme.
- An internal wildcard cert
- An internally-issued cert (say, something dropped in place by your provisioning system).
- A cert to use in case LE just failed past the expiry date.
But I see this isn't correct. Hopefully you can understand my confusion, and why I was mystified that it wasn't documented.
(To be clear, I'm not asking for it to be any of those, just explaining my thought process.)
Naming is hard. All is fine.
Closed as being stale.
From a mail to apache-users mailing list: