ACME/Let's Encrypt does not work with split horizon DNS.

[str8edgedave]

Environment

• OS: Fedora 36, fully updated
• Kubernetes/K3s: v1.24.4+k3s1 (c3f830e9)
• AWX Operator: AWX Operator 0.28.0
• cert-manager: 1.9.1

  Description

  I am using split-horizon DNS, and would like to use Let's Encrypt/ACME steps to secure my AWX installation. I am using CloudFlare as my external DNS, with BIND9 for internal DNS. Since cert-manager uses the configuration from the host system, it is picking up my internal DNS servers to perform the DNS-01 validation against.

Step to Reproduce

1. Deployed certmanager using the cert-manager yaml as per instructions. kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.9.1/cert-manager.yaml

2. configured issuers.yaml for Cloud Flare:

   
   apiVersion: cert-manager.io/v1
   kind: Issuer
   metadata:
   name: awx-issuer
   spec:
   acme:
   # The email address to be registered with ACME CA
   email: dyaffe@gmail.com
   
   # The URL of the ACME API endpoint.
   # In Let's Encrypt, this is one of the following:
   #   Production: https://acme-v02.api.letsencrypt.org/directory
   #   Staging   : https://acme-staging-v02.api.letsencrypt.org/directory
   server: https://acme-staging-v02.api.letsencrypt.org/directory
   
   privateKeySecretRef:
     name: awx-issuer-account-key
   
   solvers:
   - dns01:
       cloudflare:
         apiTokenSecretRef:
           name: cloudflare-api-token-secret
           key: client-secret```

3. configured kustomization.yaml as per documentation on cert-manager.io and here:

   
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   namespace: awx

generatorOptions: disableNameSuffixHash: true

secretGenerator:

• name: cloudflare-api-token-secret type: Opaque literals:
  • client-secret=

resources:

• issuer.yaml

  
  4. check DNS to see that the _acme-challenge.awx.<domain> has propagated:
  ```$ dig @1.1.1.1 _acme-challenge.awx.<domain> txt

; <<>> DiG 9.16.31-RH <<>> @1.1.1.1 _acme-challenge.awx. txt ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42270 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;_acme-challenge.awx.. IN TXT

;; ANSWER SECTION: _acme-challenge.awx.. 120 IN TXT ""

;; Query time: 41 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Mon Sep 05 16:11:08 CST 2022 ;; MSG SIZE rcvd: 118

At this point, cert-manager just hangs. According to the cert-manager documentation, if split-horizon DNS is being used cert-manager needs to be invoked with some extra parameters to control the DNS servers. See [https://cert-manager.io/docs/configuration/acme/dns01/#setting-nameservers-for-dns01-self-check](https://cert-manager.io/docs/configuration/acme/dns01/#setting-nameservers-for-dns01-self-check). In my case, I would have to use:
```--dns01-recursive-nameservers-only --dns01-recursive-nameservers=108.162.194.66:53,108.162.195.190:53```

[str8edgedave]

see https://github.com/cert-manager/cert-manager/issues/5415

[kurokobo]

@str8edgedave Hi, thanks for filling the issue.

First, my guide is just aimed at prividing the simplest example implementation, so the detail of how to use cert-manager is a bit out of scope of my repo. Thanks for understanding.

I would have to use: --dns01-recursive-nameservers-only --dns01-recursive-nameservers=108.162.194.66:53,108.162.195.190:53

F.Y.I., you can edit arg parameter for the Deployment resource for cert-manager bykubectl edit, or downloading cert-manager.yaml and appending args then apply it instead of kubectl appy using YAML file from URL directly. Or, of course you can use helm to install cert-manager on K3s.

[str8edgedave]

Thank-you very much!!!

I'm just getting started with Kubernetes, and with your suggestions I was able to figure it out. I just had to use the kubectl edit command, then delete the stale Certificate, order and challenge and it worked!

On suggestion for your documentation. In the issuer.yaml, you have the staging ACME api. You may want to add text about using the production issuer once everything is working with staging.

I really appreciate all you do for the community.