pdns: can't obtain certificates with lego >=4.12

[sylvainOL]

Welcome

• [X] Yes, I'm using a binary release within 2 latest releases.
• [X] Yes, I've searched similar issues on GitHub and didn't find any.
• [X] Yes, I've included all information below (version, config, etc).

What did you expect to see?

successful use of lego like (the one I've got with lego 4.11:

```console $ lego --accept-tos --email --dns pdns --server https://acme-staging-v02.api.letsencrypt.org/directory --domains test.tld --dns.disable-cp run --no-bundle 2023/07/20 04:08:43 No key found for account . Generating a P256 key. 2023/07/20 04:08:43 Saved key to /builds/oln/nif/cd/tools/lego-vault/.lego/accounts/acme-staging-v02.api.letsencrypt.org//keys/.key 2023/07/20 04:08:44 [INFO] acme: Registering account for !!!! HEADS UP !!!! Your account credentials have been saved in your Let's Encrypt configuration directory at "/builds/oln/nif/cd/tools/lego-vault/.lego/accounts". You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained from Let's Encrypt so making regular backups of this folder is ideal. 2023/07/20 04:08:44 [INFO] [test.tld] acme: Obtaining SAN certificate 2023/07/20 04:08:44 [INFO] [test.tld] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7375036434 2023/07/20 04:08:44 [INFO] [test.tld] acme: Could not find solver for: tls-alpn-01 2023/07/20 04:08:44 [INFO] [test.tld] acme: Could not find solver for: http-01 2023/07/20 04:08:44 [INFO] [test.tld] acme: use dns-01 solver 2023/07/20 04:08:44 [INFO] [test.tld] acme: Preparing to solve DNS-01 2023/07/20 04:08:47 [INFO] [test.tld] acme: Trying to solve DNS-01 2023/07/20 04:08:47 [INFO] [test.tld] acme: Checking DNS record propagation using [1.2.3.4:53] 2023/07/20 04:09:47 [INFO] Wait for propagation [timeout: 2m0s, interval: 1m0s] 2023/07/20 04:09:54 [INFO] [test.tld] The server validated our request 2023/07/20 04:09:54 [INFO] [test.tld] acme: Cleaning DNS-01 challenge 2023/07/20 04:09:57 [INFO] [test.tld] acme: Validations succeeded; requesting certificates 2023/07/20 04:09:57 [INFO] Wait for certificate [timeout: 30s, interval: 500ms] 2023/07/20 04:09:58 [INFO] [test.tld] Server responded with a certificate. ```

What did you see instead?

I've got  {"msg":"Request is not json"} when requesting pdns API (I've set here LEGO_DEBUG_CLIENT_VERBOSE_ERROR env var to TRUE in order to see a bit more:

```console $ lego --accept-tos --email --dns pdns --server https://acme-staging-v02.api.letsencrypt.org/directory --domains test.tld --dns.disable-cp run --no-bundle 2023/08/03 07:35:30 No key found for account . Generating a P256 key. 2023/08/03 07:35:30 Saved key to /builds/oln/nif/cd/tools/lego-vault/.lego/accounts/acme-staging-v02.api.letsencrypt.org//keys/.key 2023/08/03 07:35:30 [INFO] acme: Registering account for !!!! HEADS UP !!!! Your account credentials have been saved in your Let's Encrypt configuration directory at "/builds/oln/nif/cd/tools/lego-vault/.lego/accounts". You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained from Let's Encrypt so making regular backups of this folder is ideal. 2023/08/03 07:35:30 [INFO] [test.tld] acme: Obtaining SAN certificate 2023/08/03 07:35:31 [INFO] [test.tld] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7576123184 2023/08/03 07:35:31 [INFO] [test.tld] acme: Could not find solver for: tls-alpn-01 2023/08/03 07:35:31 [INFO] [test.tld] acme: Could not find solver for: http-01 2023/08/03 07:35:31 [INFO] [test.tld] acme: use dns-01 solver 2023/08/03 07:35:31 [INFO] [test.tld] acme: Preparing to solve DNS-01 2023/08/03 07:35:32 [INFO] [test.tld] acme: Cleaning DNS-01 challenge 2023/08/03 07:35:33 [WARN] [test.tld] acme: cleaning up failed: unexpected status code: [request: PUT https://pdns.server/api/v1/servers/localhost/zones/tld./notify] [status code: 400] body: {"msg":"Request is not json"} 2023/08/03 07:35:33 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7576123184 2023/08/03 07:35:33 Could not obtain certificates: error: one or more domains had a problem: [test.tld] [test.tld] acme: error presenting token: unexpected status code: [request: PUT https://pdns.server/api/v1/servers/localhost/zones/tld./notify] [status code: 400] body: {"msg":"Request is not json"} ```

How do you use lego?

Binary

Reproduction steps

1. I'm using a specific container with the following thing:

```Docker FROM alpine:3.18.2@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 ARG CI_COMMIT_SHORT_SHA ARG BUILD_DATE # renovate: datasource=repology depName=alpine_3_18/ansible-core versioning=loose ENV ANSIBLE_CORE_VERSION=2.14.5-r0 # renovate: datasource=repology depName=alpine_3_18/ca-certificates versioning=loose ENV CA_CERTIFICATES_VERSION=20230506-r0 # MITIGATE CVE-2022-3996 # to be remove when base image is fixed # renovate: datasource=repology depName=alpine_3_18/libcrypto3 versioning=loose ENV LIBCRYPTO3_VERSION=3.1.1-r3 # renovate: datasource=repology depName=alpine_3_18/libssl3 versioning=loose ENV LIBSSL3_VERSION=3.1.1-r3 LABEL org.opencontainers.image.source="https://gitlab.tld/cd/tools/lego-vault" \ org.opencontainers.image.created=$BUILD_DATE \ org.opencontainers.image.ref.name="lego-vault" \ org.opencontainers.image.authors="Sylvain Desbureaux " \ org.opencontainers.image.revision=$CI_COMMIT_SHORT_SHA SHELL ["/bin/ash", "-eox", "pipefail", "-c"] # install build deps RUN apk add --no-cache \ ansible-core="${ANSIBLE_CORE_VERSION}" \ ca-certificates="${CA_CERTIFICATES_VERSION}" \ libcrypto3="${LIBCRYPTO3_VERSION}" \ libssl3="${LIBSSL3_VERSION}" && \ rm -rf /var/cache/apk/* # hadolint ignore=DL3022 COPY --from=goacme/lego:v4.13.3 /lego /usr/bin/lego # hadolint ignore=DL3022 COPY --from=registry.gitlab.tld/python-libs/images/internal-certificate-docker:main /add-internal-certificates.sh /tmp/add-internal-certificates.sh RUN /tmp/add-internal-certificates.sh CMD [ "/bin/ash" ] ```

3. build and run in gitlab ci with the following step:

```yaml functionnal test: stage: package-test image: $DOCKER_SNAPSHOT_IMAGE variables: PDNS_API_URL: https://pdns.server PDNS_POLLING_INTERVAL: 60 LEGO_DEBUG_CLIENT_VERBOSE_ERROR: "TRUE" script: - lego --accept-tos --email --dns pdns --server https://acme-staging-v02.api.letsencrypt.org/directory --domains test.tld --dns.disable-cp run --no-bundle - ansible-vault encrypt .lego/certificates/* --vault-password-file $VAULT_PASSWORD_FILE - chmod 755 .lego - chmod 755 .lego/certificates/ - chmod 644 .lego/certificates/* rules: - if: '$CI_MERGE_REQUEST_ID' when: on_success - when: never ```

Version of lego

`v4.13.3` but the issue seems to be here since `v4.12.0`

Logs

```console $ lego --accept-tos --email --dns pdns --server https://acme-staging-v02.api.letsencrypt.org/directory --domains test.tld --dns.disable-cp run --no-bundle 2023/08/03 07:35:30 No key found for account . Generating a P256 key. 2023/08/03 07:35:30 Saved key to /builds/oln/nif/cd/tools/lego-vault/.lego/accounts/acme-staging-v02.api.letsencrypt.org//keys/.key 2023/08/03 07:35:30 [INFO] acme: Registering account for !!!! HEADS UP !!!! Your account credentials have been saved in your Let's Encrypt configuration directory at "/builds/oln/nif/cd/tools/lego-vault/.lego/accounts". You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained from Let's Encrypt so making regular backups of this folder is ideal. 2023/08/03 07:35:30 [INFO] [test.tld] acme: Obtaining SAN certificate 2023/08/03 07:35:31 [INFO] [test.tld] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7576123184 2023/08/03 07:35:31 [INFO] [test.tld] acme: Could not find solver for: tls-alpn-01 2023/08/03 07:35:31 [INFO] [test.tld] acme: Could not find solver for: http-01 2023/08/03 07:35:31 [INFO] [test.tld] acme: use dns-01 solver 2023/08/03 07:35:31 [INFO] [test.tld] acme: Preparing to solve DNS-01 2023/08/03 07:35:32 [INFO] [test.tld] acme: Cleaning DNS-01 challenge 2023/08/03 07:35:33 [WARN] [test.tld] acme: cleaning up failed: unexpected status code: [request: PUT https://pdns.server/api/v1/servers/localhost/zones/tld./notify] [status code: 400] body: {"msg":"Request is not json"} 2023/08/03 07:35:33 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7576123184 2023/08/03 07:35:33 Could not obtain certificates: error: one or more domains had a problem: [test.tld] [test.tld] acme: error presenting token: unexpected status code: [request: PUT https://pdns.server/api/v1/servers/localhost/zones/tld./notify] [status code: 400] body: {"msg":"Request is not json"} ```

Go environment (if applicable)

N/A

[ldez]

Hello,

If I create a PR to fix the problem, can you test it?

[ldez]

can you try PR #1982 ?

[sylvainOL]

I'll make a test, yes :)

[sylvainOL]

It doesn't seem to work :(

Here's what I did (I'm not sure it's the right thing...):

1. clone your PR:

```yaml retrieve fix git: stage: build image: bitnami/git:2.41.0 tags: - graas - k8s artifacts: paths: - git_source/ script: - mkdir -p git_source - cd git_source - git clone --depth 1 --single-branch --no-tags -b fix/pdns-notify https://github.com/ldez/lego.git ```

2. make lego:

```yaml build lego: stage: test image: golang:buster script: - mkdir -p dist - cd git_source/lego - make build - cp dist/lego ../../dist artifacts: paths: - dist/ ```

3. change Dockefile in order to use the created binary:

```Dockerfile FROM alpine:3.18.2@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 ARG CI_COMMIT_SHORT_SHA ARG BUILD_DATE # renovate: datasource=repology depName=alpine_3_18/ansible-core versioning=loose ENV ANSIBLE_CORE_VERSION=2.14.5-r0 # renovate: datasource=repology depName=alpine_3_18/ca-certificates versioning=loose ENV CA_CERTIFICATES_VERSION=20230506-r0 # MITIGATE CVE-2022-3996 # to be remove when base image is fixed # renovate: datasource=repology depName=alpine_3_18/libcrypto3 versioning=loose ENV LIBCRYPTO3_VERSION=3.1.1-r3 # renovate: datasource=repology depName=alpine_3_18/libssl3 versioning=loose ENV LIBSSL3_VERSION=3.1.1-r3 LABEL org.opencontainers.image.source="https://gitlab.tld/cd/tools/lego-vault" \ org.opencontainers.image.created=$BUILD_DATE \ org.opencontainers.image.ref.name="lego-vault" \ org.opencontainers.image.authors="Sylvain Desbureaux " \ org.opencontainers.image.revision=$CI_COMMIT_SHORT_SHA SHELL ["/bin/ash", "-eox", "pipefail", "-c"] # install build deps RUN apk add --no-cache \ ansible-core="${ANSIBLE_CORE_VERSION}" \ ca-certificates="${CA_CERTIFICATES_VERSION}" \ libcrypto3="${LIBCRYPTO3_VERSION}" \ libssl3="${LIBSSL3_VERSION}" && \ rm -rf /var/cache/apk/* # hadolint ignore=DL3022 COPY ./dist/lego /usr/bin/lego # hadolint ignore=DL3022 COPY --from=registry.gitlab.tld/python-libs/images/internal-certificate-docker:main /add-internal-certificates.sh /tmp/add-internal-certificates.sh RUN /tmp/add-internal-certificates.sh CMD [ "/bin/ash" ] ```

4. launch the test:

```yaml functionnal test: stage: package-test image: $DOCKER_SNAPSHOT_IMAGE variables: PDNS_API_URL: https://pdns.server PDNS_POLLING_INTERVAL: 60 LEGO_DEBUG_CLIENT_VERBOSE_ERROR: "TRUE" script: - lego --version - lego --accept-tos --email --dns pdns --server https://acme-staging-v02.api.letsencrypt.org/directory --domains test.tld --dns.disable-cp run --no-bundle - ansible-vault encrypt .lego/certificates/* --vault-password-file $VAULT_PASSWORD_FILE - chmod 755 .lego - chmod 755 .lego/certificates/ - chmod 644 .lego/certificaes/* rules: - if: '$CI_MERGE_REQUEST_ID' when: on_success - when: never ```

here's the result:

$ lego --version
lego version 52f43de48580aeb5769e7073beee4f24277be499 linux/amd64
$ lego --accept-tos --email <redacted> --dns pdns --server https://acme-staging-v02.api.letsencrypt.org/directory --domains test.gating.tld --dns.disable-cp run --no-bundle
2023/08/03 09:35:18 No key found for account <redacted>. Generating a P256 key.
2023/08/03 09:35:18 Saved key to /builds/oln/nif/cd/tools/lego-vault/.lego/accounts/acme-staging-v02.api.letsencrypt.org/<redacted>/keys/<redacted>.key
2023/08/03 09:35:19 [INFO] acme: Registering account for <redacted>
!!!! HEADS UP !!!!
Your account credentials have been saved in your Let's Encrypt
configuration directory at "/builds/oln/nif/cd/tools/lego-vault/.lego/accounts".
You should make a secure backup of this folder now. This
configuration directory will also contain certificates and
private keys obtained from Let's Encrypt so making regular
backups of this folder is ideal.
2023/08/03 09:35:19 [INFO] [test.gating.] acme: Obtaining SAN certificate
2023/08/03 09:35:20 [INFO] [test.gating.tld] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7577377814
2023/08/03 09:35:20 [INFO] [test.gating.tld] acme: Could not find solver for: tls-alpn-01
2023/08/03 09:35:20 [INFO] [test.gating.tld] acme: Could not find solver for: http-01
2023/08/03 09:35:20 [INFO] [test.gating.tld] acme: use dns-01 solver
2023/08/03 09:35:20 [INFO] [test.gating.tld] acme: Preparing to solve DNS-01
2023/08/03 09:35:21 [INFO] [test.gating.tld] acme: Cleaning DNS-01 challenge
2023/08/03 09:35:22 [WARN] [test.gating.tld] acme: cleaning up failed: unexpected status code: [request: PUT https://pdns.server/api/v1/servers/localhost/zones/tld./notify] [status code: 400] body: {"msg":"Request is not json"} 
2023/08/03 09:35:22 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7577377814
2023/08/03 09:35:22 Could not obtain certificates:
    error: one or more domains had a problem:
[test.gating.tld] [test.gating.tld] acme: error presenting token: unexpected status code: [request: PUT https://pdns.server/api/v1/servers/localhost/zones/tld./notify] [status code: 400] body: {"msg":"Request is not json"}

[sylvainOL]

I'm actually a bit stupid: I can request PDNS from my PC so I've build and test directly the PR and doesn't work either (same version as the one gave previously -- 52f43de48580aeb5769e7073beee4f24277be499-- and same result)

[ldez]

I don't understand, the doc is not really clear but I think I followed the https://doc.powerdns.com/md/httpapi/api_spec/#url-apiv1serversserver95idzoneszone95idnotify

[sylvainOL]

I'll make some tests after lunch (I've got the same behavior with curl...)

thanks!

[sylvainOL]

Ok I found the thing, you will "laugh" (or not ^_^)

Even if the body is nil, we must set Content-Type header to application/json as (almost) stated here: https://doc.powerdns.com/md/httpapi/api_spec/#data-format (it's actually false, you don't need to set Accept header but you do need Content one according to my tests...)

Previously (in <=4.11), the way to set the header was:

if method != http.MethodGet && method != http.MethodDelete {
    req.Header.Set("Content-Type", "application/json")
}

Now, it's:

if payload != nil {
    req.Header.Set("Content-Type", "application/json")
}

so I guess we should go back to previous version :)

[sylvainOL]

I can make a PR or leave it to you as you prefer!

[sylvainOL]

I've made a PR but you can make one if you prefer!

thanks again

[ldez]

All I had to do was update my PR: I asked you to test it to be able to update it if needed :wink:

[sylvainOL]

no problem, I just wanted to annoy you the least possible :)

[ldez]

There is no problem, I will keep your PR.