letsencrypt certificates on machines that share the same public hostname

[hellais]

This makes it possible for us to have the same domain name point to many different hosts, but still make it possible to automatically renew certs.

Suppose we have service.ooni.io which points to ams-srv.ooni.nu, hkg-srv.ooni.nu and mia-srv.ooni.nu (this can be done either with anycast, geoip load balancing or any other technique).

The problem is that if you try to renew the cert on ams-srv.ooni.nu you need to renew it for the common name ams-srv.ooni.nu AND service.ooni.io, but you have no guarantee which of the 3 services the acme letsencrypt scraper is going to hit.

Here is an example scenario which will break stuff:

Given the above setup, you are trying to create a certificate on host ams-srv.ooni.nu for service.ooni.io & ams-srv.ooni.nu:

1. You write the HTTP-01 challenge to the directory /var/www on ams-srv.ooni.nu
2. You run certbot to create the certificate
3. The certbot agent scrapes ams-srv.ooni.nu via http://ams-srv.ooni.nu/.well-known/acme-challenge/RANDOM_NONCE and get's an OK response (it's the actual host)
4. The certbot agent scrapes service.ooni.io via http://service.ooni.io/.well-known/acme-challenge/RANDOM_NONCE

Since service.ooni.io is mapped to many different hosts (ams-srv.ooni.nu, hkg-srv.ooni.nu and mia-srv.ooni.nu) you may hit hkg-srv.ooni.nu which will not have RANDOM_NONCE inside of it's /var/www directory and therefore it will fail.

Some useful links on letsencrypt forums are: https://community.letsencrypt.org/t/how-should-we-manage-same-certs-on-multiple-servers/85495/8 https://community.letsencrypt.org/t/letsencrypt-and-anycast/29501/4 https://community.letsencrypt.org/t/advice-on-using-letsencrypt-in-a-load-balancing-setup-ssl-passthrough/60104/2

[hellais]

It's useful to note that the letsencrypt scraper is configured to follow redirects.

A proposed solution to this problem is that we configure all the machines that are part of this "anycast group" (I will now call them like this independently from which specific load balancing technique is used) as follows:

• Each host also has a aWS token that allows to the ooni-acme-helper s3 bucket
• When they start the certbot challenge process they will write the challenge to the s3 bucket ooni-acme-helper
• Each host is also configured so that when they receive a request for /var/www/ACME_CHALLENGE they will redirect to https://ooni-acme-helper.s3.amazonaws.com/challenge/ACME_CHALLENGE

The problem with this approach is that we have to provision each machine with this secret (see: https://github.com/ooni/sysadmin/issues/364), but it does get away from having to do more complicated coordination amongst the boxes.