Fix auto-SSL domains not getting renewed

[GUI]

We have a couple agency subdomains that are managed with our automatic SSL setup that haven't been renewed on the expected schedule. The certs are still valid, but one cert will expire in 18 days, the other in 26 days. Both of these should have already been renewed automatically.

After looking at logs, it looks like our renewal script has been timing out during the renewal process. While I'm not completely certain why the renewal has been timing out, there have been several fixes in the lua-resty-auto-ssl library since we last upgraded our servers. One of the issues was also related to the timeout not being as long as expected (https://github.com/GUI/lua-resty-auto-ssl/issues/11#issuecomment-265650064). So we should upgrade lua-resty-auto-ssl, which I think will probably address this.

[GUI]

The 2 domains are now renewed, so we should be all set (and we should have gotten a fixed notification from the new monitoring setup in https://github.com/18F/api.data.gov/issues/379).

Although, the issue was potentially a bit more nuanced, so in case we see some of these same issues crop up again, here's some more detailed notes:

• One of the domains wasn't being renewed because we had an oversight in our Mongo adapter for lua-resty-auto-ssl that resulted in only the first 10 domains being renewed (and we just recently went over that limit). Here was the quick (and not particularly elegant) fix to bump the limit way up: https://github.com/18F/api.data.gov-ops/commit/0b0068d553853e32148faaa9d20a8cdfa285590e
• After updating lua-resty-auto-ssl (https://github.com/18F/api.data.gov-ops/commit/fc75b9d025523b7b31a804d4d9304849a673ad15), api.nal.usda.gov still wan't being renewed. But after the update we started to get much more useful error messages. The issue was that we were getting this error back from Let's Encrypt: DNS problem: query timed out looking up CAA for usda.gov. Things did just suddenly renew successfully, but I spent some time debugging this, so here's a few notes for reference in case this pops up again:
  • CAA record queries for usda.gov do appear to timeout (eg, dig +trace -t type257 usda.gov times out for me), so this might point to some issues with USDA's name servers. I'm not entirely sure about all this, but here was a similar thread about DNS servers that timed out handling CAA records: https://community.letsencrypt.org/t/is-there-any-way-to-pass-validation-if-dns-server-does-not-respond-to-caa/27791 The verdict of that discussion seemed to be you'd either need to fix the DNS servers or you couldn't use Let's Encrypt. So if we do see this again, we might need to discuss with USDA.
  • Here's a separate thread that notes that sometimes CAA routing can fail depending on the specific routing used: https://community.letsencrypt.org/t/cname-does-not-solve-caa-timeout-but-probably-should/32391/5 So maybe it eventually succeeded after repeated attempts because it got a different route? The lua-resty-auto-ssl with the increased timeout may have also helped get this working after repeated attempts (since it has been trying to renew for several days without success now).
  • I did change some DNS settings on our end that may have played a role. The changes I made didn't seem to fix things immediately, but if DNS caches were in play, then it's possible these helped resolve things.
  • While debugging things, I discovered that this tool considered our CNAME setup to have an error: http://dnsviz.net/d/api.nal.usda.gov/WP_RIw/dnssec/ While I don't think we've heard of any user issues, this doesn't like our nal.usda.domains.api.data.gov CNAME, since the ancestor subdomains of usda.domains.api.data.gov and domains.api.data.gov didn't have valid DNS entries themselves.
  • I added DNS entries for those ancestor subdomains to our system to resolve the warnings. Again, this didn't seem to have an immediate impact, but if we see this happen again, we might try adding other missing ancestor DNS entries (or maybe we should do that anyway).
  • We might also want to reconsider our naming convention for agency subdomains (of prepending the agency subdomain as-is to ".domains.api.data.gov"), and instead prepend a hyphenated version of the agency subdomain (so we aren't introducing a bunch of ancestor subdomains).

[GUI]

Re-opening this, since this cropped up again for our USDA.gov subdomains (but this time the issue is consistently happening, so it's a bit easier to track down).

For a brief summary and background of what's happening:

• We use Let's Encrypt for SSL on our agency subdomains, and Let's Encrypt performs CAA validation on the domain before issuing a certificate. CAA is a special type of DNS record that can be used to control which certificate authorities should be allowed to register SSL certs for your domain.
• Most agency domains don't have any CAA records, and this is fine (the absence of CAA DNS records means any certificate authority can be use, so our usage of Let's Encrypt works fine).
• However, if the CAA lookup times out, then this is interpreted as a failure (since the CAA status is unknown), and the SSL certificate cannot be registered.
• The specific issue we're facing is that we can't renew the SSL certificate for api.ers.usda.gov, since the CAA lookup that eventually checks the root usda.gov domain times out.

Let's Encrypt has a useful page outlining the ins and outs of these CAA records: https://letsencrypt.org/docs/caa/ They note timeout issues are likely an issue with the DNS's authoritative name servers:

Sometimes CAA queries time out. That is, the authoritative name server never replies with an answer at all, even after multiple retries. Most commonly this happens when your nameserver has a misconfigured firewall in front of it that drops DNS queries with unknown qtypes. File a support ticket with your DNS provider and ask them if they have such a firewall configured.

After some testing, the issue does seem that CAA queries against usda.gov's name servers time out for some reason. So while ideally this could be resolved on usda.gov's end, I'm not sure how feasible getting all that fixed is.

But luckily, we now have a way to sidestep this issue, since we can now handle the CAA lookups on our end. As Let's Encrypt mentions, CAA validation follows CNAMEs:

CAA validation follows CNAMEs, like all other DNS requests. If www.community.example.com is a CNAME to web1.example.net, the CA will first request CAA records for www.community.example.com, then seeing that there is a CNAME for that domain name instead of CAA records, will request CAA records for web1.example.net instead.

So this means we can setup CAA records on our end for the domains that each agency CNAMES to. This wasn't possible until a couple days ago when Route 53 added support for CAA records, but luckily the timing of all this worked out, and adding CAA records on our end does indeed solve this and allow us to renew SSL certificates for usda.gov subdomains.

[GUI]

As noted above, since we can now add CAA records in Route 53, so I think this should really be fixed now. We can now sidestep any CAA misconfiguration issues on agency's root domains by defining CAA records for our CNAME domain. I've added additional documentation on adding these CAA records as part of our subdomain setup, so this should be taken care of moving forward: https://github.com/18F/api.data.gov-ops#setting-up-agency-subdomains And as a bonus, defining CAA records for all of our CNAME domains is a good idea anyway (both from a security perspective and in case agencies start adding CAA records of their own).

And one small thing to note mainly for reference is that we need to add CAA records to each of our agency-specific DNS records. We can't just add a single CAA record to something like domains.api.data.gov, since Let's Encrypt does not implement "tree-climbing" to check parent domains for the underlying CNAMEd value:

The CAA RFC specifies an additional behavior called “tree-climbing” that requires CAs to also check the parent domains of the result of CNAME resolution. Let’s Encrypt does not implement tree climbing because it makes expressing certain CAA policies impossible. After discussion on the IETF mailing list, we achieved consensus that tree-climbing in CAA is not ideal, and there’s an erratum for the CAA RFC removing it.