nginx: ERROR: Cert does not exist (cause: Too many requests to LetsEncrypt!)

[ximon18]

Krill E2E test builds are failing with errors in the NGINX Docker container logs:

nginx               | Obtaining a new certificate
nginx               | An unexpected error occurred:
nginx               | There were too many requests of a given type :: Error creating new order :: too many certificates already issued for: nlnetlabs.nl: see https://letsencrypt.org/docs/rate-limits/
nginx               | Please see the logfiles in /var/log/letsencrypt for more details.
nginx               | IMPORTANT NOTES:
nginx               |  - Your account credentials have been saved in your Certbot
nginx               |    configuration directory at /etc/letsencrypt. You should make a
nginx               |    secure backup of this folder now. This configuration directory will
nginx               |    also contain certificates and private keys obtained by Certbot so
nginx               |    making regular backups of this folder is ideal.
nginx               | ERROR: Cert does not exist! Please see the validation error above. The issue may be due to incorrect dns or port forwarding settings. Please fix your settings and recreate the container

[ximon18]

The LetsEncrypt limits page states:

The main limit is Certificates per Registered Domain (50 per week). A registered domain is, generally speaking, the part of the domain you purchased from your domain name registrar. For instance, in the name www.example.com, the registered domain is example.com. In new.blog.example.co.uk, the registered domain is example.co.uk. We use the Public Suffix List to calculate the registered domain.

I was assuming, wrongly, that any limit would be avoided by using dynamically generated sub-domains but as the limit applies per registered domain this will not work. Instead it would be better to apply for a wildcard/SAN cert and install that each time.

[ximon18]

Also note:

If you’ve hit a rate limit, we don’t have a way to temporarily reset it. You’ll need to wait until the rate limit expires after a week.

[ximon18]

Using a self-signed or Lets Encrypt staging certificate was not expected to work with some Relying Parties, hence the use of a real certificate. To double check I re-ran the tests using use_staging_cert=true when deploying. The result was:

Relying Party	Test Result
Routinator	PASS
OctoRPKI	FAIL
Rcynic	FAIL
RPKI Client	FAIL
RIPE NCC RPKI Validator 3	PASS

Logs below:

module.post.null_resource.run_tests[0] (local-exec): test_krill.sh: Try 1/12: test_compare_krill_roas_to_logs routinator:
module.post.null_resource.run_tests[0] (local-exec): OKAY
module.post.null_resource.run_tests[0] (local-exec): TEST FAIL: octorpki ROAs do not match those of Krill
module.post.null_resource.run_tests[0] (local-exec): TEST FAIL: fortvalidator ROAs do not match those of Krill
module.post.null_resource.run_tests[0] (local-exec): TEST FAIL: rcynic ROAs do not match those of Krill
module.post.null_resource.run_tests[0] (local-exec): TEST FAIL: rpkiclient ROAs do not match those of Krill
module.post.null_resource.run_tests[0] (local-exec): test_krill.sh: Try 1/24: test_compare_krill_roas_to_url rpki-validator-3 http://x18openapitesting.do.nlnetlabs.nl:8080/api/export.json:
module.post.null_resource.run_tests[0] (local-exec): OKAY

[ximon18]

As a temporary workaround the Krill E2E Test was configured in commit https://github.com/NLnetLabs/krill/commit/17e234774d45fb5a19da2ef186a0dcc81b8ab8f0 to use domain krill.cloud instead of do.nlnetlabs.nl, which is not rate limited yet by Lets Encrypt. As at least some of the RPs being tested do not support (nor have a config option for supporting) self-signed/non-default-authority-signed certificates we do indeed need a real certificate. The tests should be modified to use a Lets Encrypt wildcard certificate instead of a new certificate for each test.

[ximon18]

Rather than use a wildcard LE cert that must be regularly revered by something or someone, and which requires manipulation of DO DNS records, instead use a custom CA issued cert. See branch own_ca.

[ximon18]

Fixed by 1396c5a9202625ba7914126b4f7c1359c1c00434.