Cannot create Challenge Record for internal subdomain

[aDivineDragoness]

I don't know if this is the right place or better placed somewhere upstream.

Current Situation I have my domain "domain.tld" on desec. (redacted my actual domain in this issue for privacy) For internal use, I have "local.domain.tld" on a local dns server.

The Problem Requesting a certificate for domain.tld or any other subdomain other than local.domain.tld works perfectly fine. But when I try to request a certificate for any subdomain under local.domain.tld, it doesn't create an _acme-challenge record and fails subsequently.

If it is relevant, there is a wildcard on the public domain (*.domain.tld).

The Logs:

2024/09/02 12:33:03.622 INFO    tls.obtain      obtaining certificate   {"identifier": "*.local.domain.tld"}
2024/09/02 12:33:03.623 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["*.local.domain.tld"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": ""}
2024/09/02 12:33:03.623 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["*.local.domain.tld"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": ""}
2024/09/02 12:33:03.623 INFO    tls.issuance.acme       using ACME account      {"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/161697303", "account_contact": []}
2024/09/02 12:33:03.628 INFO    tls     storage cleaning happened too recently; skipping for now        {"storage": "FileStorage:/root/.local/share/caddy", "instance": "848d2cf1-a5a8-4e02-94da-f37e7f83da53", "try_again": "2024/09/03 12:33:03.628", "try_again_in": 86399.999999642}
2024/09/02 12:33:03.629 INFO    tls     finished cleaning storage units
2024/09/02 12:33:04.769 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "*.local.domain.tld", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2024/09/02 12:33:25.006 ERROR   tls.issuance.acme.acme_client   cleaning up solver      {"identifier": "*.local.domain.tld", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.local.domain.tld\" (usually OK if presenting also failed)"}
2024/09/02 12:33:25.191 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "*.local.domain.tld", "issuer": "acme-staging-v02.api.letsencrypt.org-directory", "error": "[*.local.domain.tld] solving challenges: presenting for challenge: adding temporary record for zone \"local.domain.tld.\": writing RRSets: unexpected status code 404: {\"detail\":\"Not found.\"} (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/161697303/18818780313) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024/09/02 12:33:25.191 ERROR   tls.obtain      will retry      {"error": "[*.local.domain.tld] Obtain: [*.local.domain.tld] solving challenges: presenting for challenge: adding temporary record for zone \"local.domain.tld.\": writing RRSets: unexpected status code 404: {\"detail\":\"Not found.\"} (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/161697303/18818780313) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 21.568763446, "max_duration": 2592000}

My Config:

{
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}
test.local.domain.tld {
                tls {
                        dns desec {
                                token "token"
                        }
                        propagation_delay 300s
                }
        respond "Hello World"
}

Caddy Version: v2.8.4

I can issue a cert using acme.sh, so I doubt I misunderstand something fundamental about DNS.

[znkr]

I don't know for sure what's going on here, but I suspect that you need to create a zone for local.domain.tld.

[aDivineDragoness]

I don't know for sure what's going on here, but I suspect that you need to create a zone for local.domain.tld.

You mean on desec? I tried that and in that situation the module did not successfully create the _acme-challenge TXT records. I think either the module or the desec API cannot properly handle a zone that is a subdomain of another zone.

If you mean on my local dns server; I already have a zone there.

Either way, other tools like acme.sh have no problem. So I suspect some unnecessary checks/validations are being done.

[znkr]

adding temporary record for zone \"local.domain.tld.\": writing RRSets: unexpected status code 404: {\"detail\":\"Not found.\"}

means that local.domain.tld is used as the zone for your request. The response means that the zone doesn't exist. You said that you tried adding a zone. Did that fail in the same way, or in a different way?

[aDivineDragoness]

You said that you tried adding a zone. Did that fail in the same way, or in a different way?

In contrary to my previous try, the module was able to create the txt records this time, so that may have been previously an error on my end.

(I'm going to not redact domain this time for convencience.)

Sep 16 18:01:54 web01 caddy[3644]: {"level":"info","ts":1726502514.2776566,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/161682943","account_contact":[]}
Sep 16 18:01:54 web01 caddy[3644]: {"level":"info","ts":1726502514.2837684,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"4de23565-e7be-40b2-b703-9338c47028c7","try_again":1726588914.2837667,"try_again_in":86399.999999547}
Sep 16 18:01:54 web01 caddy[3644]: {"level":"info","ts":1726502514.2840612,"logger":"tls","msg":"finished cleaning storage units"}
Sep 16 18:01:55 web01 caddy[3644]: {"level":"info","ts":1726502515.3680348,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.local.der.gs","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Sep 16 18:13:56 web01 caddy[3644]: {"level":"error","ts":1726503236.8059382,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test.local.der.gs","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[test.local.der.gs] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/161682943/19156300723) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Sep 16 18:13:56 web01 caddy[3644]: {"level":"error","ts":1726503236.8065546,"logger":"tls.obtain","msg":"will retry","error":"[test.local.der.gs] Obtain: [test.local.der.gs] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/161682943/19156300723) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":722.529579387,"max_duration":2592000}
Sep 16 18:14:56 web01 caddy[3644]: {"level":"info","ts":1726503296.8068588,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"test.local.der.gs"}
Sep 16 18:14:56 web01 caddy[3644]: {"level":"info","ts":1726503296.807419,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/161682943","account_contact":[]}
Sep 16 18:14:57 web01 caddy[3644]: {"level":"info","ts":1726503297.3830533,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.local.der.gs","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Sep 16 18:26:58 web01 caddy[3644]: {"level":"error","ts":1726504018.9719977,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test.local.der.gs","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[test.local.der.gs] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/161682943/19156539563) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Sep 16 18:26:58 web01 caddy[3644]: {"level":"error","ts":1726504018.9727123,"logger":"tls.obtain","msg":"will retry","error":"[test.local.der.gs] Obtain: [test.local.der.gs] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/161682943/19156539563) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":1504.695736627,"max_duration":2592000}

$ dig @1.1.1.1 _acme-challenge.test.local.der.gs txt +short
"O2BjT5YqBvgn5bgahEuAPfbY78DVY38QvQpTZ92fuOg"

I have intentionally set the propagation_delay really high, just to make sure that propagation isn't the issue.

---

[...] means that local.domain.tld is used as the zone for your request

I don't like this behavior. Having to create zones for lower level subdomains makes everything more complicated. But again, i'm unsure if this is correctly placed here or better somewhere upstream.

[znkr]

[...] means that local.domain.tld is used as the zone for your request

I don't like this behavior. Having to create zones for lower level subdomains makes everything more complicated. But again, i'm unsure if this is correctly placed here or better somewhere upstream.

IIRC, we get the zone information from Caddy, this module doesn't have any control over it. So I think that needs to be a Caddy issue.

[aDivineDragoness]

oh, forgot to specifically mention it: while the TXT records now get created, it ultimately still doesn't work for me.

[znkr]

I think I remember now where Caddy gets the zone from: I think it uses the SOA record. With that, both of the failures you describe would be consistent with Caddy using a local DNS server that overrides the local.domain.tld zone. The first one, because your local DNS server would have a different zone than desec. The second one, because it's polling for the TXT record on the wrong server.

[aDivineDragoness]

That is more or less what my suspicion was.

---

Ok, now I feel dumb :woman_facepalming: Now that I have a better grasp about how caddy detects the zone and had a few better keywords to search the upstream repos, I realized that I totally missed the "resolvers"-Option in the tls directive. This can fix both cases.

I should just have RTFM more thouroughly...

Still, thank you very much for the help @znkr.

(a way to set it in the acme_dns option / globally instead of the tls directive would be nice tho :thinking: )