possible issue with wildcard DCV record

[kmei3560]

Hello! I apologize if this is not the correct repo impacted here. I have an issue that I believe is related to this library:

I have an inhouse-developed batch processing app that allows us to standardize processing of potentially large batches of public certificates.

I noticed in my last few batches that there is an issue when certificates such as the below are BOTH specified in the same batch. The issue is related to the DCV challenge record, I believe.

For example - both requested in sequence one after the other: *.example.eu.domain.app example.eu.domain.app

From my perspective, there are potentially 4 different libraries that could be involved: acmez, certmagic, libdns/route53 or the code I've added for the batch processing.

Could you please briefly review this log output I captured and see if you may have an "Aha!" moment? If not... I'm happy to dig deeper on my code assembly and see if I can narrow down the issue. Also I'm currently using an older version of libdns/route53 due to a possible issue with the latest version.

(As a sidenote: I use multiple libdns providers but it seems I occasionally run into a variance with the provider outputs. I've thought about writing a way to validate these outputs. Ideally all providers should accept the same inputs and produce identical-format outputs. A tool could help certify the providers as equivalent.)

2023/08/30 11:56:21 ca.go:236: Issue certificate #5/7 from Digicert: *.example.eu.domain.app [route53]
1.693418181555955e+09   info    obtain  acquiring lock  {"identifier": "*.example.eu.domain.app"}
1.6934181815623481e+09  info    obtain  lock acquired   {"identifier": "*.example.eu.domain.app"}
1.6934181815624409e+09  info    obtain  obtaining certificate   {"identifier": "*.example.eu.domain.app"}
1.693418181639694e+09   info    waiting on internal rate limiter    {"identifiers": ["*.example.eu.domain.app"], "ca": "https://acme.digicert.com/v2/acme/directory/", "account": "noc@example.com"}
1.693418181639717e+09   info    done waiting on internal rate limiter   {"identifiers": ["*.example.eu.domain.app"], "ca": "https://acme.digicert.com/v2/acme/directory/", "account": "noc@example.com"}
1.69341818190844e+09    info    acme_client trying to solve challenge   {"identifier": "*.example.eu.domain.app", "challenge_type": "dns-01", "ca": "https://acme.digicert.com/v2/acme/directory/"}
1.693418194070356e+09   error   acme_client cleaning up solver  {"identifier": "*.example.eu.domain.app", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.example.eu.domain.app\" (usually OK if presenting also failed)"}
1.6934181940704799e+09  info    acme_client authorization finalized {"identifier": "*.example.eu.domain.app", "authz_status": "valid"}
1.6934181940705001e+09  info    acme_client validations succeeded; finalizing order {"order": "https://acme.digicert.com/v2/acme/order/__redacted__/LxJadUh43yM1kfS9fUA3UQixZbc-dfbWNCOFxlDlOn0"}
1.693418229275489e+09   info    acme_client successfully downloaded available certificate chains    {"count": 1, "first_url": "https://acme.digicert.com/v2/acme/cert/__redacted__/12CAC69ADA26CE504BA8C09F7F60AA523E434265"}
1.6934182292897239e+09  info    obtain  certificate obtained successfully   {"identifier": "*.example.eu.domain.app"}
1.693418229289839e+09   info    obtain  releasing lock  {"identifier": "*.example.eu.domain.app"}
1.693418229464159e+09   warn    stapling OCSP   {"error": "no OCSP stapling for [*.example.eu.domain.app]: parsing OCSP response: ocsp: error from server: unauthorized", "identifiers": ["*.example.eu.domain.app"]}

2023/08/30 11:57:09 ca.go:236: Issue certificate #6/7 from Digicert: example.eu.domain.app [route53]
1.693418229483319e+09   info    obtain  acquiring lock  {"identifier": "example.eu.domain.app"}
1.693418229489676e+09   info    obtain  lock acquired   {"identifier": "example.eu.domain.app"}
1.693418229489938e+09   info    obtain  obtaining certificate   {"identifier": "example.eu.domain.app"}
1.693418229590105e+09   info    waiting on internal rate limiter    {"identifiers": ["example.eu.domain.app"], "ca": "https://acme.digicert.com/v2/acme/directory/", "account": "noc@example.com"}
1.693418229590131e+09   info    done waiting on internal rate limiter   {"identifiers": ["example.eu.domain.app"], "ca": "https://acme.digicert.com/v2/acme/directory/", "account": "noc@example.com"}
1.693418229828408e+09   info    acme_client trying to solve challenge   {"identifier": "example.eu.domain.app", "challenge_type": "dns-01", "ca": "https://acme.digicert.com/v2/acme/directory/"}
1.693418230415124e+09   error   acme_client cleaning up solver  {"identifier": "example.eu.domain.app", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.example.eu.domain.app\" (usually OK if presenting also failed)"}
1.6934182304475899e+09  error   acme_client deactivating authorization  {"identifier": "example.eu.domain.app", "authz": "https://acme.digicert.com/v2/acme/authz/__redacted__/R1sy_q1Yqiee_-vI", "error": "attempt 1: https://acme.digicert.com/v2/acme/authz/__redacted__/R1sy_q1Yqiee_-vI: HTTP 400 urn:ietf:params:acme:error:malformed - payload field of the JWS object must be the empty string"}
1.693418230447774e+09   error   obtain  could not get certificate from issuer   {"identifier": "example.eu.domain.app", "issuer": "acme.digicert.com-v2-acme-directory", "error": "[example.eu.domain.app] solving challenges: presenting for challenge: adding temporary record for zone \"eu.example.app.\": InvalidChangeBatch: operation error Route 53: ChangeResourceRecordSets, https response error StatusCode: 400, RequestID: bf7444c3-887f-40ea-b785-71f239086bb6, InvalidChangeBatch: [Tried to create resource record set [name='_acme-challenge.example.eu.domain.app.', type='TXT'] but it already exists] (order=https://acme.digicert.com/v2/acme/order/__redacted__/EpJ5lOMdCripOFtpT4rhyc8SbxJq7fALZMMbcLSQ8sc) (ca=https://acme.digicert.com/v2/acme/directory/)"}
1.693418230447844e+09   info    obtain  releasing lock  {"identifier": "example.eu.domain.app"}
2023/08/30 11:57:10 logic.go:82: example.eu.domain.app: obtaining certificate: [example.eu.domain.app] Obtain: [example.eu.domain.app] solving challenges: presenting for challenge: adding temporary record for zone "eu.example.app.": InvalidChangeBatch: operation error Route 53: ChangeResourceRecordSets, https response error StatusCode: 400, RequestID: bf7444c3-887f-40ea-b785-71f239086bb6, InvalidChangeBatch: [Tried to create resource record set [name='_acme-challenge.example.eu.domain.app.', type='TXT'] but it already exists] (order=https://acme.digicert.com/v2/acme/order/__redacted__/EpJ5lOMdCripOFtpT4rhyc8SbxJq7fALZMMbcLSQ8sc) (ca=https://acme.digicert.com/v2/acme/directory/)

[mholt]

Tried to create resource record set [name='_acme-challenge.example.eu.domain.app.', type='TXT'] but it already exists happens because the challenge record name for *.example.eu.domain.app is the same as the challenge record name for example.eu.domain.app -- but they will have different values.

Are you using CertMagic's DNS01Solver? It has this doc:

Note that challenges may be solved concurrently by some clients (such as acmez, which CertMagic uses), meaning that multiple TXT records may be created in a DNS zone simultaneously, and in some cases distinct TXT records may have the same name. For example, solving challenges for both example.com and *.example.com create a TXT record named _acme_challenge.example.com, but with different tokens as their values. This solver distinguishes between different records with the same name by looking at their values. DNS provider APIs and implementations of the libdns interfaces must also support multiple same-named TXT records.

So libdns/route53 needs to make sure it's operating on the right record resources.

If there's a bug in acmez, I'd like to be able to reproduce it otherwise I can't really fix it. Can you provide a minimal reproducer (just a few lines of code as minimal as possible) and then I can take a look?

[mholt]

Closing due to inactivity.