search

go-acme / lego

Let's Encrypt/ACME client and library written in Go
https://go-acme.github.io/lego/
MIT License
7.46k stars 986 forks source link

ionos: DNS records not removed #2082

Closed TTomczek closed 5 months ago

TTomczek commented 5 months ago

Welcome

What did you expect to see?

Removal of the created _acme_challenge DNS records after successful validation.

What did you see instead?

After the successful validation of the dns01-challange the created TXT _acme_challange records are not removed. Even though the logs state "[INFO] [traefik.example.com] acme: Cleaning DNS-01 challenge" After waiting two hours the IONOS web ui still shows the records. The same procedure with certbot/dns-ionos removes the records immediately.

How do you use lego?

Through Traefik

Reproduction steps

  1. Start docker container with the following docker compose file:
version: '3'

services:
  traefik:
    image: traefik:v2.10
    restart: unless-stopped
    volumes:
      - "letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    environment:
      - TZ=Europe/Berlin
      - TRAEFIK_API=true
      - TRAEFIK_API_DISABLEDASHBOARDAD=true
      - TRAEFIK_GLOBAL_SENDANONYMOUSUSAGE=false
      - TRAEFI_PROVIDERS_DOCKER=true
      - TRAEFIK_PROVIDERS_DOCKER_EXPOSEDBYDEFAULT=false
      - TRAEFIK_PROVIDERS_DOCKER_NETWORK=proxy
      - TRAEFIK_PROVIDERS_DOCKER_WATCH=true
      - TRAEFIK_CERTIFICATESRESOLVERS_LE=true
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_CASERVER=https://acme-staging-v02.api.letsencrypt.org/directory
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_CERTIFICATESDURATION=2160
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_DNSCHALLENGE=true
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_DNSCHALLENGE_DELAYBEFORECHECK=10
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_STORAGE=/letsencrypt/acme.json
      - TRAEFIK_LOG=true
      - TRAEFIK_LOG_LEVEL=DEBUG
      - TRAEFIK_ACCESSLOG=true
      - TRAEFIK_ENTRYPOINTS_WEB=true
      - TRAEFIK_ENTRYPOINTS_WEBSECURE=true
      - TRAEFIK_ENTRYPOINTS_WEB_ADDRESS=:80
      - TRAEFIK_ENTRYPOINTS_WEBSECURE_ADDRESS=:443
      - TRAEFIK_ENTRYPOINTS_WEB_HTTP_REDIRECTIONS_ENTRYPOINT_TO=websecure
      - TRAEFIK_ENTRYPOINTS_WEBSECURE_HTTP_TLS=true
      - TRAEFIK_ENTRYPOINTS_WEBSECURE_HTTP_TLS_CERTRESOLVER=le
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_DNSCHALLENGE_PROVIDER=ionos
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_DNSCHALLENGE_RESOLVERS=ns1***.ui-dns.com
      - IONOS_API_KEY=<API_KEY>
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_EMAIL=me@example.com
    ports:
      - 80:80
      - 443:443
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.api.rule=Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.api.service=api@internal"
      - "traefik.http.routers.api.middlewares=auth"
      - "traefik.http.routers.api.tls=true"
      - "traefik.http.routers.api.tls.certresolver=le"

volumes:
  letsencrypt:
  1. Wait for certificate creation
  2. Check web ui

Version of lego

The lego command is not available in the traefik docker image. I am using the following image.

Output of traefik version:

Version: 2.10.7
Codename: saintmarcelin
Go version: go1.21.5
Built: 2023-12-06T15:54:59Z
OS/Arch: linux/arm64

Logs

```console traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=info msg="Configuration loaded from environment variables." traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=info msg="Traefik version 2.10.7 built on 2023-12-06T15:54:59Z" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"websecure\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"certResolver\":\"le\"}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"proxy\",\"swarmModeRefreshSeconds\":\"15s\"}},\"api\":{\"dashboard\":true,\"disableDashboardAd\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"le\":{\"acme\":{\"email\":\"tim@example.com\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"ionos\",\"delayBeforeCheck\":\"10s\",\"resolvers\":[\"ns1086.ui-dns.com\"]}}}}}" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=info msg="Starting provider aggregator aggregator.ProviderAggregator" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Starting TCP Server" entryPointName=web traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Starting TCP Server" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=info msg="Starting provider *traefik.Provider" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="*traefik.Provider provider configuration: {}" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=info msg="Starting provider *docker.Provider" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"proxy\",\"swarmModeRefreshSeconds\":\"15s\"}" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=info msg="Starting provider *acme.Provider" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="*acme.Provider provider configuration: {\"email\":\"tim@example.com\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"ionos\",\"delayBeforeCheck\":\"10s\",\"resolvers\":[\"ns1086.ui-dns.com\"]},\"ResolverName\":\"le\",\"store\":{},\"TLSChallengeProvider\":{},\"HTTPChallengeProvider\":{}}" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=info msg="Starting provider *acme.ChallengeTLSALPN" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}" traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Attempt to renew certificates \"720h0m0s\" before expiry and check every \"24h0m0s\"" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=info msg="Testing certificate renew..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"web-to-websecure\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"redirect-web-to-websecure\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483646}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"redirect-web-to-websecure\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}},\"models\":{\"websecure\":{\"tls\":{\"certResolver\":\"le\"}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Provider connection established with docker 24.0.7 (API 1.43)" providerName=docker traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" providerName=docker container=pihole-exporter-pihole-exporter-e2ab7ff46cb96dd131a4bd8bba7052e0d8490fd46ac548fd8fb1a2470bccb46d traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" providerName=docker container=wireguard-wireguard-226646bff38fadef2bbe4aa7aeb5a5c08454f16053d0ab0cf1de7089e01cc5d7 traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" container=prometheus-prometheus-66e034b0bcb3cde15bc86c8ff54c057f59b23862a34f27538996343f5f55148a providerName=docker traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" providerName=docker container=watchtower-watchtower-586fd7ee6d9ad3f01b1ca17fbbd01bb947ee04b1a4bba4fe9656759c2d2fbc4d traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" providerName=docker container=homepage-homepage-31e296ed376de742b603ea22286361e0129f12afa9c0a5439312f6b7cd58e018 traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" providerName=docker container=node-exporter-node-exporter-daf8a990960e96be9760a6cf6d4113be9f5f0a1cc0a283aeb3cbeefde60484fb traefik-traefik-1 | time="2024-01-12T17:42:34+01:00" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"middlewares\":[\"auth\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik.example.com`) \\u0026\\u0026 (PathPrefix(`/api`) || PathPrefix(`/dashboard`))\",\"tls\":{\"certResolver\":\"le\"}},\"grafana\":{\"entryPoints\":[\"websecure\"],\"service\":\"grafana\",\"rule\":\"Host(`grafana.example.com`)\",\"tls\":{\"certResolver\":\"le\"}},\"pihole\":{\"entryPoints\":[\"websecure\"],\"service\":\"pihole\",\"rule\":\"Host(`dns.example.com`)\",\"tls\":{\"certResolver\":\"le\"}}},\"services\":{\"grafana\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.0.4:3000\"}],\"passHostHeader\":true}},\"pihole\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.0.2:80\"}],\"passHostHeader\":true}},\"traefik-traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.0.3:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"auth\":{\"basicAuth\":{\"users\":[\"api:$2a$10$HIV8gT9hoCe9Kgb.PbuBPeQX.NYFFHmhX8T7J6obFBPtikxXmNpqK\"]}}}},\"tcp\":{},\"udp\":{}}" providerName=docker traefik-traefik-1 | time="2024-01-12T17:42:36+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default traefik-traefik-1 | time="2024-01-12T17:42:36+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=web traefik-traefik-1 | time="2024-01-12T17:42:36+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal traefik-traefik-1 | time="2024-01-12T17:42:36+01:00" level=debug msg="Setting up redirection to https 443" routerName=web-to-websecure@internal entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme traefik-traefik-1 | time="2024-01-12T17:42:36+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery traefik-traefik-1 | time="2024-01-12T17:42:36+01:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=api traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareType=TracingForwarder middlewareName=tracing entryPointName=web routerName=web-to-websecure@internal traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Setting up redirection to https 443" middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web routerName=web-to-websecure@internal traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareType=Recovery middlewareName=traefik-internal-recovery traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=websecure routerName=websecure-api@docker middlewareName=tracing middlewareType=TracingForwarder traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" entryPointName=websecure routerName=websecure-api@docker middlewareName=auth@docker middlewareType=BasicAuth traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding tracing to middleware" entryPointName=websecure routerName=websecure-api@docker middlewareName=auth@docker traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" serviceName=grafana middlewareName=pipelining middlewareType=Pipelining routerName=grafana@docker entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating load-balancer" entryPointName=websecure serviceName=grafana routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating server 0 http://172.31.0.4:3000" entryPointName=websecure serviceName=grafana serverName=0 routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="child http://172.31.0.4:3000 now UP" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Propagating new UP status" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Added outgoing tracing middleware grafana" routerName=grafana@docker entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" routerName=pihole@docker serviceName=pihole middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating load-balancer" routerName=pihole@docker serviceName=pihole entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating server 0 http://172.31.0.2:80" entryPointName=websecure routerName=pihole@docker serverName=0 serviceName=pihole traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="child http://172.31.0.2:80 now UP" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Propagating new UP status" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Added outgoing tracing middleware pihole" routerName=pihole@docker middlewareType=TracingForwarder middlewareName=tracing entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=web traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" entryPointName=web routerName=api@docker middlewareName=auth@docker middlewareType=BasicAuth traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=api@docker middlewareName=auth@docker traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=web traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding route for grafana.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding route for dns.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Trying to challenge certificate for domain [grafana.example.com] found in HostSNI rule" routerName=grafana@docker rule="Host(`grafana.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Trying to challenge certificate for domain [dns.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Domains [\"traefik.example.com\"] need ACME certificates generation for domains \"traefik.example.com\"." rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=websecure-api@docker traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Loading ACME certificates [traefik.example.com]..." providerName=le.acme routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"grafana.example.com\"]..." routerName=grafana@docker rule="Host(`grafana.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"dns.example.com\"]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Domains [\"dns.example.com\"] need ACME certificates generation for domains \"dns.example.com\"." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Loading ACME certificates [dns.example.com]..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Domains [\"grafana.example.com\"] need ACME certificates generation for domains \"grafana.example.com\"." rule="Host(`grafana.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="Loading ACME certificates [grafana.example.com]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=grafana@docker rule="Host(`grafana.example.com`)" traefik-traefik-1 | time="2024-01-12T17:42:37+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" traefik-traefik-1 | time="2024-01-12T17:42:45+01:00" level=debug msg="Building ACME client..." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:45+01:00" level=debug msg="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:45+01:00" level=info msg=Register... providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:45+01:00" level=debug msg="legolog: [INFO] acme: Registering account for tim@example.com" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="Using DNS Challenge provider: ionos" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Obtaining bundled SAN certificate" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Obtaining bundled SAN certificate" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Obtaining bundled SAN certificate" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10575331854" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Could not find solver for: tls-alpn-01" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Could not find solver for: http-01" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: use dns-01 solver" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Preparing to solve DNS-01" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10575331864" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Could not find solver for: tls-alpn-01" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Could not find solver for: http-01" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: use dns-01 solver" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Preparing to solve DNS-01" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10575331884" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Could not find solver for: tls-alpn-01" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Could not find solver for: http-01" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: use dns-01 solver" traefik-traefik-1 | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Preparing to solve DNS-01" traefik-traefik-1 | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Trying to solve DNS-01" traefik-traefik-1 | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Checking DNS record propagation using [ns1086.ui-dns.com:53]" traefik-traefik-1 | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Trying to solve DNS-01" traefik-traefik-1 | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Checking DNS record propagation using [ns1086.ui-dns.com:53]" traefik-traefik-1 | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Trying to solve DNS-01" traefik-traefik-1 | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Checking DNS record propagation using [ns1086.ui-dns.com:53]" traefik-traefik-1 | time="2024-01-12T17:42:50+01:00" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]" traefik-traefik-1 | time="2024-01-12T17:42:50+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:50+01:00" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]" traefik-traefik-1 | time="2024-01-12T17:42:50+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:42:50+01:00" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]" traefik-traefik-1 | time="2024-01-12T17:42:50+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:00+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:01+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:01+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:02+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:03+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:03+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:13+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:13+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:13+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:15+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:15+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:15+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:25+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:25+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:25+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:27+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:27+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:27+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | 172.31.0.1 - - [12/Jan/2024:16:43:31 +0000] "GET /admin/api.php?summaryRaw&overTimeData&topItems&recentItems&getQueryTypes&getForwardDestinations&getQuerySources&jsonForceObject&auth=bd24d470b6a8fa746351a51cf0cab81ea342403acb46443d6ec48de2b3fa0960 HTTP/1.1" 301 17 "-" "-" 1 "web-to-websecure@internal" "-" 0ms traefik-traefik-1 | time="2024-01-12T17:43:37+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:37+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:37+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Waiting for DNS record propagation." traefik-traefik-1 | time="2024-01-12T17:43:39+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:39+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:43:39+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:44:00+01:00" level=debug msg="legolog: [INFO] [dns.example.com] The server validated our request" traefik-traefik-1 | time="2024-01-12T17:44:00+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Cleaning DNS-01 challenge" traefik-traefik-1 | time="2024-01-12T17:44:01+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Validations succeeded; requesting certificates" traefik-traefik-1 | time="2024-01-12T17:44:03+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] The server validated our request" traefik-traefik-1 | time="2024-01-12T17:44:03+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Cleaning DNS-01 challenge" traefik-traefik-1 | time="2024-01-12T17:44:04+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] The server validated our request" traefik-traefik-1 | time="2024-01-12T17:44:04+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Cleaning DNS-01 challenge" traefik-traefik-1 | time="2024-01-12T17:44:06+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Validations succeeded; requesting certificates" traefik-traefik-1 | time="2024-01-12T17:44:06+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Validations succeeded; requesting certificates" traefik-traefik-1 | time="2024-01-12T17:44:14+01:00" level=debug msg="legolog: [INFO] Wait for certificate [timeout: 30s, interval: 500ms]" traefik-traefik-1 | time="2024-01-12T17:44:15+01:00" level=debug msg="legolog: [INFO] [dns.example.com] Server responded with a certificate." traefik-traefik-1 | time="2024-01-12T17:44:15+01:00" level=debug msg="Certificates obtained for domains [dns.example.com]" rule="Host(`dns.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker traefik-traefik-1 | time="2024-01-12T17:44:15+01:00" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:44:15+01:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=api traefik-traefik-1 | time="2024-01-12T17:44:15+01:00" level=debug msg="Adding certificate for domain(s) dns.example.com" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=web routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-websecure@internal middlewareType=RedirectScheme middlewareName=redirect-web-to-websecure@internal traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Setting up redirection to https 443" routerName=web-to-websecure@internal middlewareType=RedirectScheme middlewareName=redirect-web-to-websecure@internal entryPointName=web traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" serviceName=grafana middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating load-balancer" serviceName=grafana entryPointName=websecure routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating server 0 http://172.31.0.4:3000" serverName=0 entryPointName=websecure routerName=grafana@docker serviceName=grafana traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="child http://172.31.0.4:3000 now UP" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Propagating new UP status" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Added outgoing tracing middleware grafana" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining serviceName=pihole entryPointName=websecure routerName=pihole@docker traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating load-balancer" routerName=pihole@docker serviceName=pihole entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating server 0 http://172.31.0.2:80" routerName=pihole@docker serviceName=pihole serverName=0 entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="child http://172.31.0.2:80 now UP" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Propagating new UP status" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Added outgoing tracing middleware pihole" entryPointName=websecure routerName=pihole@docker middlewareName=tracing middlewareType=TracingForwarder traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Added outgoing tracing middleware api@internal" routerName=websecure-api@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" routerName=websecure-api@docker middlewareName=auth@docker middlewareType=BasicAuth entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding tracing to middleware" routerName=websecure-api@docker middlewareName=auth@docker entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=web traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" middlewareType=BasicAuth entryPointName=web routerName=api@docker middlewareName=auth@docker traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=api@docker middlewareName=auth@docker traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=web traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding route for grafana.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding route for dns.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Trying to challenge certificate for domain [grafana.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`grafana.example.com`)" routerName=grafana@docker providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"grafana.example.com\"]..." routerName=grafana@docker providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`grafana.example.com`)" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="No ACME certificate generation required for domains [\"grafana.example.com\"]." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`grafana.example.com`)" routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Trying to challenge certificate for domain [dns.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"dns.example.com\"]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)" traefik-traefik-1 | time="2024-01-12T17:44:16+01:00" level=debug msg="No ACME certificate generation required for domains [\"dns.example.com\"]." rule="Host(`dns.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker traefik-traefik-1 | time="2024-01-12T17:44:22+01:00" level=debug msg="legolog: [INFO] Wait for certificate [timeout: 30s, interval: 500ms]" traefik-traefik-1 | time="2024-01-12T17:44:23+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] Server responded with a certificate." traefik-traefik-1 | time="2024-01-12T17:44:23+01:00" level=debug msg="Certificates obtained for domains [traefik.example.com]" routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:44:23+01:00" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:44:23+01:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=api traefik-traefik-1 | time="2024-01-12T17:44:23+01:00" level=debug msg="Adding certificate for domain(s) dns.example.com" traefik-traefik-1 | time="2024-01-12T17:44:23+01:00" level=debug msg="Adding certificate for domain(s) traefik.example.com" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareType=TracingForwarder entryPointName=web routerName=web-to-websecure@internal middlewareName=tracing traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" middlewareType=RedirectScheme entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Setting up redirection to https 443" routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" serviceName=pihole middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=pihole@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=pihole@docker serviceName=pihole traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating server 0 http://172.31.0.2:80" serviceName=pihole serverName=0 entryPointName=websecure routerName=pihole@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="child http://172.31.0.2:80 now UP" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Propagating new UP status" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Added outgoing tracing middleware pihole" routerName=pihole@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=websecure-api@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" entryPointName=websecure routerName=websecure-api@docker middlewareName=auth@docker middlewareType=BasicAuth traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding tracing to middleware" middlewareName=auth@docker entryPointName=websecure routerName=websecure-api@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" routerName=grafana@docker middlewareType=Pipelining middlewareName=pipelining serviceName=grafana entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=grafana@docker serviceName=grafana traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating server 0 http://172.31.0.4:3000" serviceName=grafana serverName=0 entryPointName=websecure routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="child http://172.31.0.4:3000 now UP" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Propagating new UP status" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Added outgoing tracing middleware grafana" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=api@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" routerName=api@docker middlewareName=auth@docker middlewareType=BasicAuth entryPointName=web traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=api@docker middlewareName=auth@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareType=Recovery middlewareName=traefik-internal-recovery traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=web traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding route for dns.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding route for grafana.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." providerName=le.acme rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Trying to challenge certificate for domain [grafana.example.com] found in HostSNI rule" rule="Host(`grafana.example.com`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"grafana.example.com\"]..." rule="Host(`grafana.example.com`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Trying to challenge certificate for domain [dns.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=pihole@docker rule="Host(`dns.example.com`)" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=websecure-api@docker traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="No ACME certificate generation required for domains [\"grafana.example.com\"]." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=grafana@docker rule="Host(`grafana.example.com`)" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"dns.example.com\"]..." providerName=le.acme routerName=pihole@docker rule="Host(`dns.example.com`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:44:24+01:00" level=debug msg="No ACME certificate generation required for domains [\"dns.example.com\"]." providerName=le.acme routerName=pihole@docker rule="Host(`dns.example.com`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | 172.31.0.1 - - [12/Jan/2024:16:44:31 +0000] "GET /admin/api.php?summaryRaw&overTimeData&topItems&recentItems&getQueryTypes&getForwardDestinations&getQuerySources&jsonForceObject&auth=bd24d470b6a8fa746351a51cf0cab81ea342403acb46443d6ec48de2b3fa0960 HTTP/1.1" 301 17 "-" "-" 2 "web-to-websecure@internal" "-" 0ms traefik-traefik-1 | time="2024-01-12T17:44:32+01:00" level=debug msg="legolog: [INFO] Wait for certificate [timeout: 30s, interval: 500ms]" traefik-traefik-1 | time="2024-01-12T17:44:33+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] Server responded with a certificate." traefik-traefik-1 | time="2024-01-12T17:44:33+01:00" level=debug msg="Certificates obtained for domains [grafana.example.com]" rule="Host(`grafana.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:33+01:00" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:44:33+01:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=api traefik-traefik-1 | time="2024-01-12T17:44:33+01:00" level=debug msg="Adding certificate for domain(s) dns.example.com" traefik-traefik-1 | time="2024-01-12T17:44:33+01:00" level=debug msg="Adding certificate for domain(s) traefik.example.com" traefik-traefik-1 | time="2024-01-12T17:44:33+01:00" level=debug msg="Adding certificate for domain(s) grafana.example.com" traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareType=TracingForwarder entryPointName=web routerName=web-to-websecure@internal middlewareName=tracing traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" routerName=web-to-websecure@internal entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Setting up redirection to https 443" routerName=web-to-websecure@internal entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" routerName=pihole@docker serviceName=pihole middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating load-balancer" serviceName=pihole entryPointName=websecure routerName=pihole@docker traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating server 0 http://172.31.0.2:80" serverName=0 entryPointName=websecure routerName=pihole@docker serviceName=pihole traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="child http://172.31.0.2:80 now UP" traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Propagating new UP status" traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Added outgoing tracing middleware pihole" routerName=pihole@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=websecure-api@docker traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" entryPointName=websecure routerName=websecure-api@docker middlewareName=auth@docker middlewareType=BasicAuth traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding tracing to middleware" entryPointName=websecure routerName=websecure-api@docker middlewareName=auth@docker traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=grafana@docker serviceName=grafana traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating load-balancer" routerName=grafana@docker serviceName=grafana entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating server 0 http://172.31.0.4:3000" routerName=grafana@docker serviceName=grafana serverName=0 entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="child http://172.31.0.4:3000 now UP" traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Propagating new UP status" traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Added outgoing tracing middleware grafana" middlewareType=TracingForwarder entryPointName=websecure routerName=grafana@docker middlewareName=tracing traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=web routerName=api@docker middlewareName=tracing traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" routerName=api@docker middlewareName=auth@docker middlewareType=BasicAuth entryPointName=web traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=api@docker middlewareName=auth@docker traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=web traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding route for grafana.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding route for dns.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=websecure traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=websecure-api@docker traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." routerName=websecure-api@docker providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=websecure-api@docker providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Trying to challenge certificate for domain [grafana.example.com] found in HostSNI rule" rule="Host(`grafana.example.com`)" routerName=grafana@docker providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Trying to challenge certificate for domain [dns.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)" providerName=le.acme traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"grafana.example.com\"]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`grafana.example.com`)" routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="No ACME certificate generation required for domains [\"grafana.example.com\"]." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`grafana.example.com`)" routerName=grafana@docker traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"dns.example.com\"]..." routerName=pihole@docker rule="Host(`dns.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" traefik-traefik-1 | time="2024-01-12T17:44:34+01:00" level=debug msg="No ACME certificate generation required for domains [\"dns.example.com\"]." routerName=pihole@docker rule="Host(`dns.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" ```

Go environment (if applicable)

```console $ go version go1.21.5 ```
ldez commented 5 months ago

@jankatins @AlexH-HankIT can you check that?

ldez commented 5 months ago

The lego command is not available in the traefik docker image.

Traefik and lego are written in Go, it's a compiled language, so dependencies are not executable (it's not Python).

@TTomczek can you try lego directly? https://go-acme.github.io/lego/installation/

TTomczek commented 5 months ago

@ldez I tried it with the lego docker image using the following command:

docker run --rm -e IONOS_API_KEY=<API_KEY> goacme/lego --accept-tos --email="me@example.com" --dns ionos --domains="*.example.com" --server="https://acme-staging-v02.api.letsencrypt.org/directory" --dns.resolvers="ns1***.ui-dns.com" run

After i successfully requested the certificate and waiting 45 minutes the record is still there.

Logs:

```console 2024/01/14 14:45:43 No key found for account me@example.com. Generating a P256 key. 2024/01/14 14:45:43 Saved key to /.lego/accounts/acme-staging-v02.api.letsencrypt.org/me@example.com/keys/me@example.com.key 2024/01/14 14:45:44 [INFO] acme: Registering account for me@example.com !!!! HEADS UP !!!! Your account credentials have been saved in your Let's Encrypt configuration directory at "/.lego/accounts". You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained from Let's Encrypt so making regular backups of this folder is ideal. 2024/01/14 14:45:44 [INFO] [*.example.com] acme: Obtaining bundled SAN certificate 2024/01/14 14:45:44 [INFO] [*.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/***** 2024/01/14 14:45:44 [INFO] [*.example.com] acme: use dns-01 solver 2024/01/14 14:45:44 [INFO] [*.example.com] acme: Preparing to solve DNS-01 2024/01/14 14:45:47 [INFO] [*.example.com] acme: Trying to solve DNS-01 2024/01/14 14:45:47 [INFO] [*.example.com] acme: Checking DNS record propagation using [ns1***.ui-dns.com:53] 2024/01/14 14:45:49 [INFO] Wait for propagation [timeout: 1m0s, interval: 2s] 2024/01/14 14:45:49 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:45:51 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:45:53 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:45:55 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:45:57 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:45:59 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:01 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:03 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:05 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:07 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:09 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:11 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:13 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:15 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:18 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:20 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:22 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:24 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:26 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:28 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:30 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:32 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:34 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:36 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:39 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:41 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:43 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:46:45 [INFO] [*.example.com] acme: Waiting for DNS record propagation. 2024/01/14 14:47:02 [INFO] [*.example.com] The server validated our request 2024/01/14 14:47:02 [INFO] [*.example.com] acme: Cleaning DNS-01 challenge 2024/01/14 14:47:05 [INFO] [*.example.com] acme: Validations succeeded; requesting certificates 2024/01/14 14:47:05 [INFO] Wait for certificate [timeout: 30s, interval: 500ms] 2024/01/14 14:47:06 [INFO] [*.example.com] Server responded with a certificate. ```
jankatins commented 5 months ago

I can confirm this: I use traefik to create lets encrypt certs against my ionos hosted domain and I have a ton of _acme-challenge.<subdomain> in my TXT records for my domain.

ldez commented 5 months ago

@jankatins can you try my PR #2083? The PR doesn't fix the problem but it will help to diagnose.

ldez commented 5 months ago

@jankatins have you tried my PR?

ldez commented 5 months ago

@TTomczek if I explain how to build the PR, can you test it?

jankatins commented 5 months ago

Here you go:

λ git pr 2083 # checks out the PR #2083 
λ make build

# Redaced real email and domain
λ  IONOS_API_KEY="<key>" dist/lego --accept-tos --email="email@example.com" --dns ionos --domains="*.invalid.example.com" --server="https://acme-staging-v02.api.letsencrypt.org/directory"  run
2024/01/17 17:26:32 No key found for account email@example.com. Generating a P256 key.
2024/01/17 17:26:32 Saved key to /home/jan/projects/lego/.lego/accounts/acme-staging-v02.api.letsencrypt.org/email@example.com/keys/email@example.com.key
2024/01/17 17:26:33 [INFO] acme: Registering account for email@example.com
!!!! HEADS UP !!!!

Your account credentials have been saved in your Let's Encrypt
configuration directory at "/home/jan/projects/lego/.lego/accounts".

You should make a secure backup of this folder now. This
configuration directory will also contain certificates and
private keys obtained from Let's Encrypt so making regular
backups of this folder is ideal.
2024/01/17 17:26:33 [INFO] [*.invalid.example.com] acme: Obtaining bundled SAN certificate
2024/01/17 17:26:34 [INFO] [*.invalid.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10666513264
2024/01/17 17:26:34 [INFO] [*.invalid.example.com] acme: use dns-01 solver
2024/01/17 17:26:34 [INFO] [*.invalid.example.com] acme: Preparing to solve DNS-01
2024/01/17 17:26:37 [INFO] [*.invalid.example.com] acme: Trying to solve DNS-01
2024/01/17 17:26:37 [INFO] [*.invalid.example.com] acme: Checking DNS record propagation using [100.100.100.100:53]
2024/01/17 17:26:39 [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]
2024/01/17 17:26:39 [INFO] [*.invalid.example.com] acme: Waiting for DNS record propagation.
2024/01/17 17:27:04 [INFO] [*.invalid.example.com] The server validated our request
2024/01/17 17:27:04 [INFO] [*.invalid.example.com] acme: Cleaning DNS-01 challenge
2024/01/17 17:27:06 [INFO] Name: _acme-challenge.invalid.example.com, Content: "...."
2024/01/17 17:27:06 [WARN] [*.invalid.example.com] acme: cleaning up failed: ionos: failed to remove record (zone=d6e2815f-4fe7-11eb-857e-0a58644464b1, domain=invalid.example.com, fqdn=_acme-challenge.invalid.example.com., value=...): %!w(<nil>)
2024/01/17 17:27:06 [INFO] [*.invalid.example.com] acme: Validations succeeded; requesting certificates
2024/01/17 17:27:06 [INFO] Wait for certificate [timeout: 30s, interval: 500ms]
2024/01/17 17:27:07 [INFO] [*.invalid.example.com] Server responded with a certificate.

The relevant line again with added line breaks:

2024/01/17 17:27:06 [WARN] [*.invalid.example.com] acme: cleaning up failed: ionos: failed to remove record
(zone=d6e2815f-4fe7-11eb-857e-0a58644464b1, domain=invalid.example.com, 
fqdn=_acme-challenge.invalid.example.com., value=...): %!w(<nil>)
ldez commented 5 months ago

@jankatins thank you. Based on your logs, I think I found the problem: the record content/value has quotes when coming from the API ("....")

[INFO] Name: _acme-challenge.invalid.example.com, Content: "...."
domain=invalid.example.com, fqdn=_acme-challenge.invalid.example.com., value=...)

I updated the PR, can you try it?

jankatins commented 5 months ago

Looks better:

~/projects/lego on pr/2083:refs/pull/2083/head (025621a0) took 29s
[18:13:16] λ  IONOS_API_KEY="key" dist/lego --accept-tos --email="email@example.com" --dns ionos --domains="*.invalid1.example.com" --server="https://acme-staging-v02.api.letsencrypt.org/directory"  run
2024/01/17 18:13:34 [INFO] [*.invalid1.example.com] acme: Obtaining bundled SAN certificate
2024/01/17 18:13:35 [INFO] [*.invalid1.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10667079314
2024/01/17 18:13:35 [INFO] [*.invalid1.example.com] acme: use dns-01 solver
2024/01/17 18:13:35 [INFO] [*.invalid1.example.com] acme: Preparing to solve DNS-01
2024/01/17 18:13:37 [INFO] [*.invalid1.example.com] acme: Trying to solve DNS-01
2024/01/17 18:13:37 [INFO] [*.invalid1.example.com] acme: Checking DNS record propagation using [100.100.100.100:53]
2024/01/17 18:13:39 [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]
2024/01/17 18:14:03 [INFO] [*.invalid1.example.com] The server validated our request
2024/01/17 18:14:03 [INFO] [*.invalid1.example.com] acme: Cleaning DNS-01 challenge
2024/01/17 18:14:07 [INFO] Name: _acme-challenge.invalid1.example.com, Content: "...."
2024/01/17 18:14:07 [INFO] [*.invalid1.example.com] acme: Validations succeeded; requesting certificates
2024/01/17 18:14:08 [INFO] Wait for certificate [timeout: 30s, interval: 500ms]
2024/01/17 18:14:08 [INFO] [*.invalid1.example.com] Server responded with a certificate.
ldez commented 5 months ago

Thank you again, the PR is ready now.

jankatins commented 5 months ago

Just deleted 130 challenges for ~3 subdomains :-)