proxy-dns is [possibly] caching negative responses

[bsyk]

Using certbot to renew certificates using dns-01 challenge. The machine that’s running certbot client is using cloudflared as a dns-proxy for DoH.

In the process of renewing a certificate, certbot inserts a TXT record _acme-challenge.host.domain using the Cloudflare API. This record is correctly created.

Certbot then proceeds to poll for this value to ensure it’s queryable before asking the certificate provider to generate a certificate. The first query for this record results in an NXDOMAIN result since the record has not yet propagated to queryable DNS servers.

Subsequent polls, are getting a cached negative result. You can see the TTL decreasing on manual queries. The initial TTL on the negative result appears to be 30 minutes. Certbot times-out after 5 minutes of polling for the record, so certificate renewal never succeeds. Restarting the cloudflared process clears the cache and later queries successfully get the correct TXT record result.

I don’t know whether this is a change to cloudflared or whether upstream authoritative servers are now sending SOA TTLs for negative responses resulting in cloudflared caching the negative result. The effect is that using certbot with dns-01 challenge, via Cloudflare + cloudflared proxy-dns does not work.

This machine had been renewing certificates happily. The most recent renewal failed and the certificate expired. Suggesting that something changed within the last 3 months. The version of cloudflared being used is from Aug 22.

Note: I’m actually using the built-in certificate issuance from Caddy, but used certbot here as a more familiar term. Also posted here: https://community.cloudflare.com/t/cloudflared-proxy-dns-is-caching-negative-responses/414454

[cliffjao]

I seem to be running into this exact same issue. Is there a solution for it, such as dnsmasq's no-negcache option?

[bsyk]

@cliffjao my work-around has been to create a fake TXT record in the domain. This works because the certbot never gets an NXDOMAIN, but sees another unrelated result and continues to poll. Once the correct value is added, certbot sees both the real and fake entry and proceeds happily.

My fake record:

;; ANSWER SECTION:
_acme-challenge.lights.host.domain. 60 IN   TXT "placeholder"

[cliffjao]

@bsyk Interesting workaround. Thanks!

[cliffjao]

Upon further thought & research, it seems like the default convention for DNS proxies (dnsmasq, cloudflared, others) is to negative cache NXDOMAIN records. I'm guessing that's why certbot has propagation-seconds parameters so that it doesn't fetch a negative record before allowing it to propagate:

The number of seconds to wait for DNS to propagate before asking the ACME server to verify the DNS record.

I'm guessing increasing this time could also be a solution.

[alex4108]

I managed to RCA a negative caching incident to cloudflared 2024.4.1. Raised a PR above