Caddy On-demand TLS and subdomain enumerators

[mcpherrinm]

I've noticed an interesting pattern, which I'm primarily filing as a bug for your awareness, without any specific action required, and not unique to Caddy.

A common configuration is to point a DNS wildcard address at a Caddy server. DNS wildcards in servers often resolve multiple levels of labels to some IPs.

Some (probably malicious) web scanners load domains from CT and try to enumerate subdomains for all the certificates they find.

This can lead to a feedback loop where Caddy on-demand TLS issues a new subdomain, causing the subdomain enumerators to try again, repeating until the hostname length or maximum number of labels limit is hit, or a rate limit is reached.

For example, https://crt.sh/?q=git.git.git.git https://crt.sh/?q=www.www.www.www

[francislavoie]

If you're using On-Demand, it's your responsibility to block invalid domains like this using your ask endpoint. This isn't something Caddy can do for you.

[mcpherrinm]

Well, it seems to be a widespread problem in practice, and may be representative of a bad default or hard-to-configure solution. Caddy appears to be generating a fairly significant chunk of these bogus-looking domains.

It's not at a high enough level to merit taking any action on Let's Encrypt's side, but I just wanted to flag it.

[francislavoie]

Ah, didn't realize you were writing on behalf of Let's Encrypt.

Ultimately there's not much we can do here in Caddy I think; I figure you might need to rate limit users producing bad domains like this more aggressively. The ask endpoint is required when enabling On-Demand TLS, so it must be lazy users who allow all domains without checking an allow-list properly. :man_shrugging:

[emilylange]

The ask endpoint is required when enabling On-Demand TLS, so it must be lazy users who allow all domains without checking an allow-list properly. 🤷‍♂️

Or instances with old and no longer supported Caddy releases.

The ask endpoint only became mandatory as part of https://github.com/caddyserver/caddy/releases/tag/v2.7.3, which has been released roughly 7 months ago.

Before that, Caddy printed a warning about this on start-up.

[mcpherrinm]

Ah, thanks. It sounds like this is already mitigated in the current version then. I do see this happening with 2.6.* clients mostly, though there's some 2.7.3+ in there.

[mholt]

Just reading this now that I'm at my desk:

Yes, we've always had an 'ask' endpoint and required it for quite some time now. Isn't this mainly hurting the site owners though? LE rate limits should be blocking subdomain issuances pretty quickly after the first, what, 50 in the week? Enumerators will quickly run up that limit and site owners should notice their service not functioning as expected for legitimate users.

Not a good situation, obviously, but also brought upon themselves since we are very clear about the 'ask' endpoint to grant permission before certificate issuances.

Anyway, this is something we've known about for over 8 years and have had facilities in place to prevent it.

[mcpherrinm]

To be clear, it isn’t hurting Let’s Encrypt at all. I was mostly investigating why it was happening as a curiosity - I saw some funny strings combinations of “www”, “git”, and “gitlab” reoccurring in CT and spent a few moments looking into it. I’m not that familiar with Caddy so I wasn’t aware of the ask endpoint.

[mholt]

Ah, gotcha. Glad it's not a problem then. We are always trying to help keep things tidy :)