Incorrect TXT record with wildcard and non-wildcard in the same cert

[rhaamo]

I'm trying using ACMEv2 and wildcard and I have some issues. My domains.txt contains something like:

foo.domain.tld *.foo.domain.tld bar.domain.tld

Config for hook:

CHALLENGETYPE='dns-01'
HOOK='/opt/pdns_api.sh/pdns_api.sh'
HOOK_CHAIN="yes"

When deploying using pdns_api.sh dns hook I got created two entries with a different challenge:

_acme-challenge.foo TXT SOME_VALUE1
_acme-challenge.bar TXT SOME_VALUE2

But in logs I get:

Processing foo.domain.tld with alternative names: *.domain.tld bar.domain.tld
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 3 authorizations URLs from the CA
 + Handling authorization for foo.domain.tld
 + Handling authorization for bar.domain.tld
 + Handling authorization for foo.domain.tld
 + 3 pending challenge(s)
 + Deploying challenge tokens...
 + Responding to challenge for foo.domain.tld authorization...
ERROR: Challenge is invalid! (returned: invalid) (result: {
  "type": "dns-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:unauthorized",
    "detail": "Incorrect TXT record \"SOME_VALUE1\" found at _acme-challenge.foo.domain.tld",
    "status": 403
  },
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/xxx/xxx",
  "token": "xxx",
  "keyAuthorization": "xxx"
})

I also tried first with the production ACMEv2 and got the same issue.

I ran with debug in config file and at one point it shows:

Name:  _acme-challenge.bar.domain.tld.
Token: SOME_VALUE2
Zone:  domain.tld.
Name:  _acme-challenge.foo.domain.tld.
Token: SOME_VALUE3???
Zone:  domain.tld.
Name:  _acme-challenge.foo.domain.tld.
Token: SOME_VALUE1
Zone:  domain.tld.

I see the same thing being pushed to pdns, with obviously only the "last" challenge in the update list being accepted for foo.domain.tld.

Is it supported to have a cert with foo.domain.tld *.foo.domain.tld bar.domain.tld or there is a bug somewhere ?

(Also I can't everytime reproduce it properly because randomly I only get a challenge generated for foo.domain.tld and nothing for bar.domain.tld :( )

[txr13]

Your hook and/or DNS software may need to be updated.

dehydrated is generating the correct values for your DNS records. ACME validates a wildcard domain (*.foo.domain.tld) using the same DNS record name as for the parent non-wildcard domain (foo.domain.tld), but with a different record value. Your debug file is correctly showing you that you will need to have two different tokens uploaded, but both with the same name.

This is allowable under DNS. For example, if you look for TXT records under "google.com." you will see two of them. (One is an SPF record, the other is a Docusign record.) When validating your certificate, ACME will check for all records under the required name. One will match the wildcard validator, and the other will match the non-wildcard validator.

I don't know where the issue here lies--your hook may be calling an "update" operation in pdns which overwrites the first token, or pdns may not correctly support multiple TXT records for the same DNS name. But the issue does not lie within dehydrated.

(If you have successfully validated bar.domain.tld, you may not be getting a challenge generated for it on follow-up runs, since ACME will re-use validated identifiers without requiring reauthorization, at least within a certain time frame.)

[bviktor]

Actually it doesn't need the wildcard to break at all. I've found that it needs 3 things to be in place at the same time. If

• I use v2 endpoint (either prod or staging) AND
• Use top-level domain, i.e. foobar.com AND
• Use an alias

I get this error. The 3rd one suggests that maybe it's something messed up dehydrated itself. If any of this isn't true it works fine.

[txr13]

@bviktor If you're getting an error when using v2, validating the domain apex, and using an alias (but NOT validating a wildcard), you may want to open a separate issue. This one is specific to validating a wildcard and non-wildcard at the same time, whether there's an alias in use or not.

[bviktor]

My point is that these are most likely related to the same root problem. In fact, having

foo.com > foo.com

Can be substituted with

foo.com *.foo.com

They both result in the same exact problem, assuming the other 2 conditions are also present (of course, wildcard cert implies v2).

While

foo.com

Or

*.foo.com

Or

foo.com bar.foo.com

All work. So it seems an issue with aliases, and maybe the same mechanism is in place when a wildcard cert is present after the root domain.

[txr13]

That's not correct.

For one, multiple people report that requesting a certificate with wildcard and non-wildcard works. This issue is specific to the pdns hook.

Aliases are processed differently. Your second example of foo.com *.foo.com does not use an alias at all, so it won't be triggering whatever is causing your alias issue to occur. Aliases are not a substitute for a wildcard, and wildcards cannot be substituted for aliases.

[bviktor]

For one, multiple people report that requesting a certificate with wildcard and non-wildcard works. This issue is specific to the pdns hook.

It's definitely not specific to the pdns hook because it happens to me with the certzure hook.

Aliases are processed differently. Your second example of foo.com *.foo.com does not use an alias at all, so it won't be triggering whatever is causing your alias issue to occur.

But dehydrated needs to generate a name for it, doesn't it? It won't create a foo.com *.foo.com folder, it'll be foo.com. Just like with an alias.

[txr13]

Okay, then aliases have nothing to do with it. This issue is specifically about validating wildcards and non-wildcards in the same call.

Does the certzure hook allow you to create multiple records with the same DNS name simultaneously? Can you verify that you can see all records created via the hook in your DNS server, or are some missing?

[bviktor]

It can't create multiple records, but I don't understand why it's only a requirement for the above 3 conditions at the same time. I mean

foo.com bar.foo.com

Also requires 2 records, but dehydrated deploys and validates them one by one, be it v1 or v2. Why change this if having an alias or a toplevel + wildcard?

Also, why would you need 2 records at the same time for foo.com > foo.com? It's most certainly just one record. This "multiple records at once" reasoning doesn't add up for me. Let alone "aliases have nothing to do with it". Yes, they do. There's no multiple records, there's no wildcard, yet it still breaks. I remove the alias and it starts working.

[txr13]

This issue is for validating a wildcard and non-wildcard domain at the same time. Specifically, this is using HOOK_CHAIN=yes. Which means the hook must be capable of deploying two TXT records, both of the form _acme-challenge.foo.com. but with two different tokens. One will validate the wildcard, and one will validate the non-wildcard.

If you're validating foo.com and bar.foo.com and they are being validated one by one (instead of at the same time), then you would appear to not be using HOOK_CHAIN=yes. And even if you were, you wouldn't be uploading two records of the same name--you would have two names (one of _acme-challenge.foo.com. and one of _acme-challenge.bar.foo.com.).

If you're having troubles with an alias, that's different again. An alias is used as a directory name / certificate-specific config filename, no more and no less. It does not impact the validation records at all. So if you're experiencing an issue with an alias, it should be put in a separate issue, so it doesn't get lost in the discussion over validation records.

[lukas2511]

Yea, like @txr13 said this is an issue with the dns hook script, for hook-chaining with certificates like example.com *.example.com you'll need to have multiple txt records with corresponding challenge tokens on the same domain at the same time.

Without chaining it would in theory work, but keep in mind that Let's Encrypt caches DNS entries for up to 5 minutes, so you may run into validation issues. After validating one of the two names it doesn't need to be validated again, so only the second one gets validated, and since now there is only one record it will obviously succeed.

Best way is to just improve the dns hook-script.

[bviktor]

Is there documentation about how to do so? Currently this is what happens:

deploy_challenge A
deploy_challenge B
clean_challenge A -> valid
clean_challenge B -> invalid

Which is kinda weird to me because if deploy_challenge B overwrote A with B (which it did), then I'd assume that clean_challenge A should fail and clean_challenge B should succeed. Unless I'm missing something here.

Do I have to add the token to the same _acme-challenge record as a 2nd value? But then which clean_challenge should delete which value?

[lukas2511]

@bviktor things can get a bit confusing with two authorizations having the same human-readable name combined with dns caching and other stuff

Your zone should look something like this after all challenges have been deployed:

_acme-challenge.example.com. 300 TXT "foo"
_acme-challenge.example.com. 300 TXT "bar"

Without hook-chaining the process should look something like this:

# INFO: Using main config file /etc/dehydrated/config-v2
HOOK: this_hookscript_is_broken__dehydrated_is_working_fine__please_ignore_unknown_hooks_in_your_script
HOOK: startup_hook
Processing dehydrated.de with alternative names: *.dehydrated.de
 + Creating new directory /etc/dehydrated/certs/dehydrated.de ...
HOOK: this_hookscript_is_broken__dehydrated_is_working_fine__please_ignore_unknown_hooks_in_your_script
HOOK: generate_csr dehydrated.de /etc/dehydrated/certs/dehydrated.de dehydrated.de *.dehydrated.de
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 2 authorizations URLs from the CA
 + Handling authorization for dehydrated.de
 + Handling authorization for dehydrated.de
 + 2 pending challenge(s)
 + Deploying challenge tokens...
HOOK: deploy_challenge dehydrated.de x1 y1
HOOK_LOG: + Deploying challenge token for dehydrated.de.........
HOOK: deploy_challenge dehydrated.de x2 y2
HOOK_LOG: + Deploying challenge token for dehydrated.de.........
 + Responding to challenge for dehydrated.de authorization...
HOOK: clean_challenge dehydrated.de x1 y1
HOOK_LOG: + Removing challenge token on dehydrated.de.........
 + Challenge is valid!
 + Responding to challenge for dehydrated.de authorization...
HOOK: clean_challenge dehydrated.de x2 y2
HOOK_LOG: + Removing challenge token on dehydrated.de.........
 + Challenge is valid!
 + Requesting certificate...
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...
 + Using cached chain!
HOOK: deploy_cert dehydrated.de /etc/dehydrated/certs/dehydrated.de/privkey.pem /etc/dehydrated/certs/dehydrated.de/cert.pem /etc/dehydrated/certs/dehydrated.de/fullchain.pem /etc/dehydrated/certs/dehydrated.de/chain.pem 1521047586
 + Done!
 + Updating OCSP stapling file
HOOK: exit_hook

[bviktor]

Will try to port certzure to azure-cli v2 coz I don't feel like messing around with Java too much. WIll make testing and debugging easier too. Then I'll get back to this issue.

In the meanwhile - what's "hook-chaining" and how do I use it? Google doesn't show up much...

[lukas2511]

@bviktor dehydrated has a mode where it basically puts everything that has to be deployed to dns (for a single certificate) into a single hook-call (see https://github.com/lukas2511/dehydrated/blob/master/docs/hook_chain.md). It's especially useful for people with lots of domains, but it could also make logic for a hook a bit easier (deploy everything, delete verything, instead of deploying and removing one by one)

[jangrewe]

For those affected by this issue, there's a fix for pdns_api.sh: https://github.com/silkeh/pdns_api.sh/pull/17