Route 53 Split-horizon DNS

[itomaldonado]

When using split-horizon DNS in AWS we have two hosted zones (a private and a public one) with the same domain.

Right now the service is choosing the first one it finds (which is the private one) and LE verification times out.

Since for LE to verify your cert it needs to do a public DNS lookup, this service should default to the Public hosted-zone if it finds more than one zone with the same domain.

[itomaldonado]

I am taking a stab at it if you want to....

[tazjin]

Thanks for pointing this out, I'll take a look soon!

[tazjin]

Fixed in #64

[itomaldonado]

@tazjin Just wanted to put this here for future reference. If you are using split-brain DNS it is possible for the application to resolve the DNS entry with the built-in name servers which will return the "private hosted zone" NS entries. To force the application to use an external DNS for resolving the name spaces make sure you do add the environment variable "LETSENCRYPT_CONTROLLER_OPTS" as seen below (the example uses google's DNS server).

...
    spec:
      serviceAccountName: letsencrypt
      containers:
        - image: tazjin/letsencrypt-controller:1.7
          imagePullPolicy: Always
          name: letsencrypt-controller
          env:
          - name: LETSENCRYPT_CONTROLLER_OPTS
            value: "-Ddns.server=8.8.8.8"
...

[tazjin]

Ah, so this can cause local validation to fail before the controller asks Let's Encrypt to verify the challenge?

The way it currently works is that the controller uses the OS DNS to ask for the authoritative nameservers after performing the change and polls them until the change has propagated. It should probably be changed to resolve from the root nameservers, too. 🤔

[itomaldonado]

Yeah so when the controller should not try to use the OS DNS to ask for the authoritative nameservers but doing so is a good default behavior.

I have run into this issue several times before and there is no easy way to fix this. When using split-brain local DNS (i.e. the OS's default DNS) will resolve to the private side (it should since it is intended behavior) and thus it'll mess with the DNS validation.

So as long as there is a way to tell the controller what DNS server to use during validation, then this problem is solved.

[tazjin]

but doing so is a good default behavior

I'm thinking doing a full resolve from the DNS root may actually be better, because that is what Let's Encrypt does.

Thanks for pointing this out, I'll make a separate issue and see if there's anything that can be done to improve it.

As you mentioned overriding the dns.server property will let people apply a fix currently, I'll probably pull that into the README.

[itomaldonado]

Perfect, thanks~ Awesome work by the way.