caddyserver / caddy

Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS
https://caddyserver.com
Apache License 2.0
56.47k stars 3.96k forks source link

WildCard SSL doesnt' work. "acme: cleaning up failed: no memory of presenting a DNS record for ..." #3474

Closed hsea closed 4 years ago

hsea commented 4 years ago

1. Environment

1a. Operating system and version

Debian 4.19.37-5+deb10u2 (2019-08-08) x86_64 GNU/Linux

1b. Caddy version (run caddy version or paste commit SHA)

v2.0.0 h1:pQSaIJGFluFvu8KDGDODV8u4/QRED/OPyIR+MWYYse8=

1c. Go version (if building Caddy from source; run go version)

go version go1.14.4 linux/amd64

2. Description

2a. What happens (briefly explain what is wrong)

TLS Configure works for xxx.xx and www.xxx.xx, it also works for

xxx.xx www.xxx.xx {
    tls {
    ...
    }
}

but it doesn't work for

*.xxx.xx {
    tls {
    ...
    }
}

2b. Why it's a bug (if it's not obvious)

2c. Log output

Jun 05 02:10:04 ONEVPS200530075317 caddy[30426]: {"level":"info","ts":1591323004.6141896,"msg":"serving initial configuration"}
Jun 05 02:10:04 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:04 [INFO][cache:0xc0006bd860] Started certificate maintenance routine
Jun 05 02:10:04 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:04 [INFO][*.rseco.cf] Obtain certificate; acquiring lock...
Jun 05 02:10:04 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:04 [INFO][*.rseco.cf] Obtain: Lock acquired; proceeding...
Jun 05 02:10:05 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:05 [INFO][*.rseco.cf] Waiting on rate limiter...
Jun 05 02:10:05 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:05 [INFO][*.rseco.cf] Done waiting
Jun 05 02:10:05 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:05 [INFO] [*.rseco.cf] acme: Obtaining bundled SAN certificate given a CSR
Jun 05 02:10:06 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:06 [INFO] [*.rseco.cf] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5026703172
Jun 05 02:10:06 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:06 [INFO] [*.rseco.cf] acme: use dns-01 solver
Jun 05 02:10:06 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:06 [INFO] [*.rseco.cf] acme: Preparing to solve DNS-01
Jun 05 02:10:06 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:06 [INFO] [*.rseco.cf] acme: Preparing to solve DNS-01
Jun 05 02:10:06 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:06 [INFO] [*.rseco.cf] acme: Cleaning DNS-01 challenge
Jun 05 02:10:06 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:06 [WARN] [*.rseco.cf] acme: cleaning up failed: no memory of presenting a DNS record for rseco.cf
Jun 05 02:10:07 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:07 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5026703172
Jun 05 02:10:07 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:07 [ERROR] error: one or more domains had a problem:
Jun 05 02:10:07 ONEVPS200530075317 caddy[30426]: [*.rseco.cf] [*.rseco.cf] acme: error presenting token: got error status: HTTP 401: []
Jun 05 02:10:07 ONEVPS200530075317 caddy[30426]:  (challenge=dns-01 remaining=[])
Jun 05 02:10:09 ONEVPS200530075317 caddy[30426]: 2020/06/05 02:10:09 [ERROR] attempt 1: [*.rseco.cf] Obtain: [*.rseco.cf] error: one or more domains had a problem:
Jun 05 02:10:09 ONEVPS200530075317 caddy[30426]: [*.rseco.cf] [*.rseco.cf] acme: error presenting token: got error status: HTTP 401: []
Jun 05 02:10:09 ONEVPS200530075317 caddy[30426]:  - retrying in 1m0s (4.745202486s/720h0m0s elapsed)...

2d. Workaround(s)

xcaddy build --with github.com/caddy-dns/cloudflare

Already check the token but it still doesn't work. https://github.com/libdns/cloudflare

3. Tutorial (minimal steps to reproduce the bug)

Caddyfile

*.rseco.cf {
        tls {
                 dns cloudflare {env.CLOUDFLARE_API_TOKEN}
       }
        @www {
                host www.rseco.cf
        }
        reverse_proxy @www localhost
}

rseco.cf {
        log {
                output file /var/log/caddy/caddy.log
        }
        root * /var/www/html
                encode zstd gzip
                file_server
}

caddy.service

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
Environment=CLOUDFLARE_EMAIL="my email"   
Environment=CLOUDFLARE_API_TOKEN="my taken"        

[Install]
WantedBy=multi-user.target
mholt commented 4 years ago

Thanks for opening an issue, but unfortunately you've redacted all the important bits. We'll need to know the exact domain names used in order to look into this more.

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! It will ask for some information you've already provided; that's OK, just fill it out the best you can. :+1:

I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```

### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```

### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```

## 2. Description

### 2a. What happens (briefly explain what is wrong)

### 2b. Why it's a bug (if it's not obvious)

### 2c. Log output

```
paste terminal output or logs here
```

### 2d. Workaround(s)

### 2e. Relevant links

## 3. Tutorial (minimal steps to reproduce the bug)

Helpful tips

  1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

  2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

    • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
    • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
    • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.
  3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

    • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
    • Do not run Caddy inside containers.
    • Run Caddy manually in your terminal; do not use systemd or other init systems.
    • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
    • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
    • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file: ``` { ... } ``` Open terminal and run Caddy: ``` $ caddy ... ``` Make an HTTP request: ``` $ curl ... ``` Notice that the result is ___ but it should be ___.
hsea commented 4 years ago

@mholt

Thank you for the promote reply. 👍 I modified the description above. I wrote a minimum Caddyfile to reproduce the bug in my host and added the complete output except the credentials. If email and token of cloudflare is necessary, feel free to leave a message.

mholt commented 4 years ago

Thank you! The credentials are probably not necessary, I will look into this when I get a chance.

mholt commented 4 years ago

After drilling down a bit, it seems that our upstream dependency, lego, has a code path that calls CleanUp without having called Present (sometimes?), when using the DNS challenge.

Mon-ius commented 4 years ago

I have the same problem too. Since the code and documents are all bad. I look through the forum and GitHub issues, there is no way to fix it, even no none-direct way. This problem here for me only happens when using caddy2 with Cloudflare compiled by xcaddy. The caddy2 installed via apt doesn't have the problem for me. In this time, the caddy2 even doesn't support go-1.5, which sucks.

Here is the solution for me which is a little bit complicated but works:

Solution

Turn the Proxy status into DNS only

with the normal one:


    tls YOUR_EMAIL
    tls {
        dns cloudflare YOUR_CF_KEY
    }

Turn back Proxy status into Proxied

How I find the solution?

To use caddy2 with Cloudflare, I follow the documents and find xcaddy. Then, xcaddy needs go installed, actually higher go version installed. The default one via apt is 1.1, then updated to the lastest one 1.5. However, it is too hight for caddy2. Not go-1.5, have to downgrade to go-1.4.

Then finally for me, to use caddy2 with Cloudflare. Everything works well at that time, so, I follow the document: https://caddyserver.com/docs/conventions#file-locations, and try to find where the new location is. But I soon cannot find where are my key and cert. Where these two guys think it's still good. https://caddy.community/t/caddy-v2-location-of-certificatefile-and-keyfile/9486/6. Now I can see why the problem still not be fixed for months.

Follow the logs, I find out where it is. And get to know why it appears on a wired location. (/var/lib/caddy, No mentioned on documents.) The caddy2 I installed via apt specify the USER caddy on the /lib/systemd/system/caddy.service. So I removed the caddy user on /lib/systemd/system/caddy.service. Then I meet the same problem WildCard SSL doesnt' work. "acme: cleaning up failed: no memory of presenting a DNS record for ...".

I checked back the old location for the cert and key file. I surprisedly find that there are key and cert already been generated, without error. Then I tried to find the solution, and figure out the ugly way but works.

mholt commented 4 years ago

Since we finished the transition away from lego, this should be fixed with the latest iterations of CertMagic and ACMEz. It's unreleased still since Caddy 2.2 is blocked on a dependency to be compatible with Go 1.15, but if you build the latest from source and make sure to use the latest versions of Caddy, CertMagic, and ACMEz, it should work like a charm.

francislavoie commented 4 years ago

Or you can target v2.2.0-rc.1 rather than master, when building with xcaddy.

Mon-ius commented 4 years ago

run xcaddy build v2.2.0-rc.1 --with github.com/caddy-dns/cloudflare@latest and replace the /usr/bin/caddy

still the same error, challenge_type": "dns-01", "error": "no memory of presenting a DNS record for. Not sure if need to rebuild xcaddy as well.

❯ caddy version
v2.2.0-rc.1 h1:ULGBB9efRs8yG27IBOMLljhCa4iM1IluFTDxjm+Y8vE=
mholt commented 4 years ago

Thanks. No you don't need to rebuild xcaddy. I think I know why this is happening... it's subtle. It has to do with trying to get certs for both rseco.cf and *.rseco.cf. The challenge domain for both is rseco.cf which may be problematic since we solve these in parallel.

(By the way, I saw a report from another user the other day that Cloudflare's API doesn't work with .cf domains so you might be out of luck even once we get this working.)

francislavoie commented 4 years ago

More info about the blocked domains: https://community.cloudflare.com/t/freenom-free-domains-cf-partners-and-cf-dns/187631

Mon-ius commented 4 years ago

I noticed that the original caddy2 from apt will not generate and use *.domain.com as v1 does. I tried to add some subdomain. It will produce, sub1.domain.com.(key, crt) , sub2.domain.com.(key, crt), etc.

mholt commented 4 years ago

@hsea @Mon-ius This should now be fixed (it works for me), but I'd like you to confirm if possible.

You'll need to build from source again to verify: xcaddy build c94f5bb7dd3c98d6573c44f06d99c7252911a9fa --with github.com/caddyserver/certmagic@c2f851df75f0c52144a91e00ac53110034865faf --with github.com/caddy-dns/cloudflare

I think that ^ command will do the trick...

Mon-ius commented 4 years ago

Not working as well.


ERROR   tls.issuance.acme.acme_client   cleaning up solver  {"identifier": "xxxx", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for xxxxx (probably OK if presenting failed)"}
2020/08/19 16:39:09.982 ERROR   tls.obtain  will retry  {"error": "[xxxxx] Obtain: [xxxxx] solving challenges: presenting for challenge: adding temporary record for zone xxxxx.: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme-v02.api.letsencrypt.org/acme/order/xxxx) (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 4.282310513, "max_duration": 2592000}
mholt commented 4 years ago

"Invalid request headers" is an error sent by the cloudflare server, double check your configuration and credentials I guess.

Mon-ius commented 4 years ago

I don't think that is my problem. I tried to use the same configuration under Cloudflare Website. And Caddy v1 and Caddy v2, v1 did work.

mholt commented 4 years ago

@Mon-ius I have precedent to believe it is: https://caddy.community/t/cloudflare-dns-challenge-for-tls-some-subdomains-work-others-fail-dns-challenge/9110/5?u=matt

Caddy 1 and Caddy 2 are not compatible. We're now using a totally new, custom ACME stack. You'll need to make sure to use the right credentials in your config.

Mon-ius commented 4 years ago

I understand, v1 use email, v2 use API_TOKEN. But I don't think that is the reason why v2 doesn't work.

I can ensure that I use the scoped token not global api.

Mon-ius commented 3 years ago

I found an interesting thing that, .com domain will work but another domain, like .tk, .ml, .xyz, etc. May not. Maybe you can use the old feature on v1 to make all domain works on v2.

francislavoie commented 3 years ago

As we said, that's not something we can change. That's Cloudflare blocking those domains. https://github.com/caddyserver/caddy/issues/3474#issuecomment-675228355