TXT record created with malformed name when using subdomain under a DNS zone

[PTaylour]

I've been using the GoDaddy webhook with cert-manager on an on-prem cluster and I've encountered a situation that I'm hoping to get some clarification on.

We're running an internal DNS server with a specified DNS zone (zone.mycompany.org)

When I request a certificate for a sub-subdomain (e.g., app.zone.mycompany.org), the GoDaddy webhook seems to attempt to create the TXT record under zone.mycompany.org (as expected), but GoDaddy ends up creating the record under mycompany.org instead.

The PUT request made by the webhook:

PUT https://api.godaddy.com/v1/domains/zone.mycompany.org/records/TXT/_acme-challenge.app

results in the record being accessible at _acme-challenge.app.mycompany.org (zone is missing)

dig -t txt _acme-challenge.app.mycompany.org +short
"c-JuZ0H6dcsr1oVTk0INFciyviEi42Fnvn_mTxf5ErF"

and not at _acme-challenge.app.zone.mycompany.org

dig -t txt _acme-challenge.app.zone.mycompany.org +short

as I would expect.

I can work around this by telling cert-manager to use only public DNS servers:

dns01RecursiveNameservers: "1.1.1.1:53,1.0.0.1:53"
dns01RecursiveNameserversOnly: true

Which results in the PUT request

PUT https://api.godaddy.com/v1/domains/mycompany.org/records/TXT/_acme-challenge.app.zone

and creates the TXT record at _acme-challenge.app.zone.mycompany.org as required.

However, I'm wondering if this implies there's a limitation or a specific configuration requirement when using DNS zones with the GoDaddy webhook? Or is this likely to be an issue with how we've configured the internal DNS server?

Thanks for your time and assistance!

Additional information

Here's my config

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: me@mycompany.org
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
    - selector:
        dnsNames:
        - app.zone.mycompany.org
      dns01:
        webhook:
          config:
            apiKeySecretRef:
              name: godaddy-api-key
              key: token
            production: true
            ttl: 1800
          groupName: acme.mycompany.com
          solverName: godaddy
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-cert
spec:
  secretName: ingress-cert-secret
  commonName: app.zone.mycompany.org
  dnsNames:
    - app.zone.mycompany.org
  duration: 2160h
  renewBefore: 360h
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer

[cmoulliard]

Here's my config

Can you also share the ChallengeRequest created by the Cert Manager as the ACME webhook extracts the information from what we got from the challenge using the following fields: https://github.com/cert-manager/cert-manager/blob/master/pkg/acme/webhook/apis/acme/v1alpha1/types.go#L81-L95 and this is such a recordName that the webhook will present using a REST call to godaddy ? @PTaylour

Remark: Can you use the main_test.go to test your configuration ?

[PTaylour]

Thanks for looking into this, I appreciate it!

Can you also share the ChallengeRequest created by the Cert Manager

Is this logged out somewhere? I've set v=6 on cert-manager and logLevel: dubug on godaddy-webhook but can't see it. Is there another way to get hold of it?

Remark: Can you use the main_test.go to test your configuration ?

Works okay if use the top domain:

make test TEST_ZONE_NAME=mycompany.org.
ok      github.com/snowdrop/godaddy-webhook     35.639s

But timesout if I use the zone:

make test TEST_ZONE_NAME=zone.mycompany.org.
time="2023-11-06T16:06:22Z" level=info msg="TXT Record name: cert-manager-dns01-tests"
time="2023-11-06T16:06:22Z" level=info msg="### Try to present the DNS record with the DNS provider using as challengeKey: 123d=="
time="2023-11-06T16:06:22Z" level=info msg="### URL request issued to check if the TXT DNS record is present: /v1/domains/zone.mycompany.org/records/TXT/cert-manager-dns01-tests"
time="2023-11-06T16:06:22Z" level=info msg="### URL request issued to create/update the DNS record: /v1/domains/zone.mycompany.org/records/TXT/cert-manager-dns01-tests"
time="2023-11-06T16:06:23Z" level=info msg="### TXT record created/updated using godaddy REST API !"
time="2023-11-06T16:09:23Z" level=info msg="### CleanUp should delete the relevant TXT record for the challengeKey: 123d=="
time="2023-11-06T16:09:23Z" level=info msg="### URL request issued to check if the TXT DNS record is present: /v1/domains/zone.mycompany.org/records/TXT/cert-manager-dns01-tests"
[controller-runtime] log.SetLogger(...) was never called, logs will not be displayed:
goroutine 216 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
        /home/ptaylour/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/log/log.go:59 +0xcd
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc000443480, 0xc00061b9b8?)
        /home/ptaylour/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/log/deleg.go:111 +0x32
github.com/go-logr/logr.Logger.Enabled(...)
        /home/ptaylour/go/pkg/mod/github.com/go-logr/logr@v1.2.4/logr.go:261
github.com/go-logr/logr.Logger.Info({{0x23625e8?, 0xc000443480?}, 0x0?}, {0x205fc9e, 0x16}, {0x0, 0x0, 0x0})
        /home/ptaylour/go/pkg/mod/github.com/go-logr/logr@v1.2.4/logr.go:274 +0x72
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start(0xc00092e280)
        /home/ptaylour/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/envtest/server.go:260 +0x419
github.com/cert-manager/cert-manager/test/apiserver.RunBareControlPlane(0xc0008d36c0)
        /home/ptaylour/go/pkg/mod/github.com/cert-manager/cert-manager@v1.12.3/test/apiserver/apiserver.go:44 +0x2f
github.com/cert-manager/cert-manager/test/acme.(*fixture).setup(0xc0003c3d40, 0xc0008d36c0)
        /home/ptaylour/go/pkg/mod/github.com/cert-manager/cert-manager@v1.12.3/test/acme/fixture.go:114 +0x105
github.com/cert-manager/cert-manager/test/acme.(*fixture).RunExtended(0xc0003c3d40, 0x0?)
        /home/ptaylour/go/pkg/mod/github.com/cert-manager/cert-manager@v1.12.3/test/acme/fixture.go:100 +0x2d
github.com/cert-manager/cert-manager/test/acme.(*fixture).RunConformance.func1(0xc000527fd0?)
        /home/ptaylour/go/pkg/mod/github.com/cert-manager/cert-manager@v1.12.3/test/acme/fixture.go:88 +0x36
testing.tRunner(0xc0008d36c0, 0xc000917040)
        /usr/local/go/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 89
        /usr/local/go/src/testing/testing.go:1648 +0x3ad
--- FAIL: TestRunsSuite (199.80s)
    --- FAIL: TestRunsSuite/Conformance (194.17s)
        --- FAIL: TestRunsSuite/Conformance/Basic (183.75s)
            --- FAIL: TestRunsSuite/Conformance/Basic/PresentRecord (183.75s)
                util.go:61: skipping file "testdata/godaddy/README.md" with unrecognised extension
                util.go:61: skipping file "testdata/godaddy/godaddy.secret.example" with unrecognised extension
                util.go:70: created fixture "basic-present-record"
                suite.go:38: Calling Present with ChallengeRequest: &v1alpha1.ChallengeRequest{UID:"", Action:"", Type:"", DNSName:"example.com", Key:"123d==", ResourceNamespace:"basic-present-record", ResolvedFQDN:"cert-manager-dns01-tests.zone.mycompany.org.", ResolvedZone:"zone.mycompany.org.", AllowAmbientCredentials:false, Config:(*v1.JSON)(0xc00035b5d8)}
                suite.go:48: error waiting for DNS record propagation: context deadline exceeded

[cmoulliard]

Can you also share the ChallengeRequest created by the Cert Manager

This is an object that you can get using "kubectl get ChallengeRequest" @PTaylour

[PTaylour]

I didn't have any luck with kubectl get ChallengeRequest

The closest resources I have to that are:

❯ kubectl api-resources | grep -i "^ch\|^ce"
challenges                                     acme.cert-manager.io/v1                true         Challenge
certificaterequests               cr,crs       cert-manager.io/v1                     true         CertificateRequest
certificates                      cert,certs   cert-manager.io/v1                     true         Certificate
certificatesigningrequests        csr          certificates.k8s.io/v1                 false        CertificateSigningRequest

I had a look at each of these and couldn't see the ResovledZone or ResolvedFQDN.

I'm on v1.3.1 of cert-manager and v0.2.0 of the webhook

❯ kubectl get deployment -n cert-manager -o=jsonpath='{$.items[*].spec.template.spec.containers[:1].image}'            
quay.io/jetstack/cert-manager-controller:v1.13.1 quay.io/jetstack/cert-manager-cainjector:v1.13.1 quay.io/jetstack/cert-manager-webhook:v1.13.1 quay.io/snowdrop/cert-manager-webhook-godaddy:0.2.0%

Thanks again for your help

[cmoulliard]

Sorry. My fault, I was referring to Challenge which is created from CertificateRequest -> order -> challenge (see: https://cert-manager.io/docs/concepts/acme-orders-challenges/)

[bordenit]

Do the dependencies here roll back up into acme.sh by chance. I had the same issue there recently, and used a wildcard instead of messing with it further. It was the same exact thing of the DNS record not getting created properly when subdomain was used so I just changed it to *.mycompany.org for my domain in this example and moved on. https://github.com/acmesh-official/acme.sh/pull/4870

[gitdr]

@PTaylour have you managed to solve your issue without using dns01RecursiveNameservers and dns01RecursiveNameserversOnly

[alexstojda]

I'm having a similar issue to this.

I have a domain example.com for which I want to issue a certificate on test.example.com.

The Challenge resource reports an error 422 in its status, which after digging a bit seems to be from the call to GoDaddy's API to check if the TXT challenge record exists or not.

The API call looks like this: GET /v1/domains/_acme-challenge.test.example.com/records/TXT/_acme-challenge.test.example.com and returns the 422. The call should be GET /v1/domains/example.com/records/TXT/_acme-challenge.test.example.com

The same workaround proposed by @PTaylour, adding the below settings to cert-manager, fixes the issue, but I don't fully understand why, nor if this is the actual solution to the issue.

dns01RecursiveNameservers: "1.1.1.1:53,1.0.0.1:53"
dns01RecursiveNameserversOnly: true