search

caddyserver / caddy

Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS
https://caddyserver.com
Apache License 2.0
57.86k stars 4.02k forks source link

Caddy sends too many requests on failed certificate request #4186

Closed MCWertGaming closed 3 years ago

MCWertGaming commented 3 years ago

Hello!

I'm running Caddy with my local smallstep CA and faced the issue that caddy flooded it with way too many certificate requests. The problem was that caddy tried to request a certificate for a non-existent domain (which was a configuration mistake). The result was that caddy started to send 8 certificate requests per second and keep going for hours (because I didn't noticed). In the end my CA crashed because it's database has gotten too big.

Shouldn't caddy limit it's requests to something like one per domain every 5 minutes? Or is there already a configuration option or something like that? ALso it does TLS, HTTP and DNS validation at the same time. Is that normal? Or can I simply limit it only use TLS?

Thank you for helping!

Best regards Damon Leven

MCWertGaming commented 3 years ago

For reference see this discussion I started on the smallstep repository of my CA: https://github.com/smallstep/certificates/discussions/598

mholt commented 3 years ago

Thanks for opening an issue! We'll look into this.

It's not immediately clear to me what is going on, so I'll need your help to understand it better.

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! This will require some effort on your part -- please understand that we will be dedicating time to fix the bug you are reporting if you can just help us understand it and reproduce it easily.

This template will ask for some information you've already provided; that's OK, just fill it out the best you can. :+1: I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```

### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```

### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```

## 2. Description

### 2a. What happens (briefly explain what is wrong)

### 2b. Why it's a bug (if it's not obvious)

### 2c. Log output

```
paste terminal output or logs here
```

### 2d. Workaround(s)

### 2e. Relevant links

## 3. Tutorial (minimal steps to reproduce the bug)

Instructions -- please heed otherwise we cannot help you (help us help you!)

  1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

  2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

    • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
    • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
    • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

  3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

    • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
    • Do not run Caddy inside containers.
    • Run Caddy manually in your terminal; do not use systemd or other init systems.
    • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
    • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
    • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file: ``` { ... } ``` Open terminal and run Caddy: ``` $ caddy ... ``` Make an HTTP request: ``` $ curl ... ``` Notice that the result is ___ but it should be ___.
francislavoie commented 3 years ago

FYI, Caddy's ACME error behaviour is documented here: https://caddyserver.com/docs/automatic-https#errors

MCWertGaming commented 3 years ago

FYI, Caddy's ACME error behaviour is documented here: https://caddyserver.com/docs/automatic-https#errors

I have already seen that page, but instead of a "brief pause" caddy does 8 requests per second over hours in this case. I'll provide more information in a few minutes. Thank you for Helping with this!

MCWertGaming commented 3 years ago

1. Environment

1a. Operating system and version

[root@ca ~]# uname -a
Linux ca 5.4.114-1-pve #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05 +0200) x86_64 x86_64 x86_64 GNU/Linux
[root@ca ~]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@ca ~]#

NOTE: Both caddy and step-ca are running inside of docker. Step-ca is running in the official alpine powered docker container available on docker hub. Caddy is running inside of an custom docker container created by myself which is basically an ubi8-minimal (RHEL8 based) image on ontop of that i'm installing my root certificate (because of my step-ca) and download the newest caddy release (binary) from github. I don't think that this causes the problem, but if you need more informations about how I'm packaging my caddy I can provide the docker file and the docker-compose file.

1b. Caddy version (run caddy version or paste commit SHA)

[root@server ~]# docker run docker.localdomain.com/caddy-trusted:latest caddy version
v2.4.1 h1:kAJ0JB5Xk5gPdTH/27S5cyoMGqD5lBAe9yZ8zTjVJa0=

1c. Go version (if building Caddy from source; run go version)

not installed as i'm not building from source.

2. Description

2a. What happens (briefly explain what is wrong)

I have my smallstep certification authority fully setup and configured on my server A. Then, I'm starting my caddy (who acts as reverse proxy) on Server B. Server B basically acts as gateway for my services so that I can expose all of my services into other sub nets of my local network wile only granting access to port 443 and 80 of a single machine. Server B has many domains, like docker.local, gitserver.local, jenkins.local. I'm now creating a Caddyfile for all those services. Without noticing I make a typo in one domain or already add an domain for later use which is not yet configured in my DNS server. As I start caddy with my local docker installation, everything works. Caddy orders certificates and exposes those services. The interesting thing is that you only see the failed certificate orders, if you enable debug logging in your Caddyfile. If it's enabled, you see that caddy is consistently trying to order certificates without even waiting. He is literally flodding Server A with acme requests. Working domains are not facing this issue.

Meanwhile on server A you can see that smallsteps nosql database starts to rapidly increases in it's size until 200mb are reached (which are probably a few thousand certificates) and step-ca dies and is not able to handle any new acme requests. A restart of server A repaires the CA for a short time, while the second restart ended up in a destroyed nosql database.

The documentations notes that caddy is retrying it one time and after the acme request fails, it sleeps for an increasing time between tries. You can clearly see that caddy is sending 8 acme requests per second! (Images can be found here smallstep/certificates#598)

2b. Why it's a bug (if it's not obvious)

Well, the sleep is not working.

2c. Log output

compose_caddy_1 is up-to-date
Attaching to compose_caddy_1
[mcaddy_1  | {"level":"info","ts":1622734099.1825364,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
[mcaddy_1  | {"level":"warn","ts":1622734099.185338,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
[mcaddy_1  | {"level":"info","ts":1622734099.188164,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
[mcaddy_1  | {"level":"info","ts":1622734099.1889353,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
[mcaddy_1  | {"level":"info","ts":1622734099.1893623,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
[mcaddy_1  | {"level":"debug","ts":1622734099.1910233,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
[mcaddy_1  | {"level":"debug","ts":1622734099.1915488,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
[mcaddy_1  | {"level":"info","ts":1622734099.1919515,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["docker.nexus.domain.local","gitea.domain.local","drone.domain.local","nexus.domain.local","docker-private.nexus.domain.local"]}
[mcaddy_1  | {"level":"info","ts":1622734099.1927474,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"}
[mcaddy_1  | {"level":"info","ts":1622734099.193118,"msg":"serving initial configuration"}
[mcaddy_1  | {"level":"info","ts":1622734099.1939464,"logger":"tls.obtain","msg":"acquiring lock","identifier":"docker.nexus.domain.local"}
[mcaddy_1  | {"level":"info","ts":1622734099.1963995,"logger":"tls.obtain","msg":"lock acquired","identifier":"docker.nexus.domain.local"}
[mcaddy_1  | {"level":"info","ts":1622734099.2017248,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00025e310"}
[mcaddy_1  | {"level":"info","ts":1622734099.202097,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/root/.local/share/caddy"}
[mcaddy_1  | {"level":"info","ts":1622734099.2024717,"logger":"tls","msg":"finished cleaning storage units"}
[mcaddy_1  | {"level":"info","ts":1622734099.2031426,"logger":"tls.obtain","msg":"acquiring lock","identifier":"gitea.domain.local"}
[mcaddy_1  | {"level":"info","ts":1622734099.2047646,"logger":"tls.obtain","msg":"lock acquired","identifier":"gitea.domain.local"}
[mcaddy_1  | {"level":"info","ts":1622734099.2055056,"logger":"tls.obtain","msg":"acquiring lock","identifier":"drone.domain.local"}
[mcaddy_1  | {"level":"info","ts":1622734099.207091,"logger":"tls.obtain","msg":"lock acquired","identifier":"drone.domain.local"}
[mcaddy_1  | {"level":"info","ts":1622734099.2078173,"logger":"tls.obtain","msg":"acquiring lock","identifier":"nexus.domain.local"}
caddy_1  | {"level":"debug","ts":1622734135.0459683,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:55 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["aDJGZGg3WDRTeEtheDhpa2Z4bVVmM01ueGlqUWlTb2Q"]}}
caddy_1  | {"level":"debug","ts":1622734135.3049612,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:55 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["MVF5ckdHVGszQW11OGJpOUIxR0YwQjZVamFZNE9DUlU"]}}
caddy_1  | {"level":"debug","ts":1622734135.305728,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:55 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["SGlYRXcxTkNxUkkzZzNiaVdYOTdWY0xHalNJdzU0T3A"]}}
caddy_1  | {"level":"debug","ts":1622734135.5648255,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:55 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["TDlOTUJUdG44elFKbnlabThPbVl1VVZYbFM3UkFJVmg"]}}
caddy_1  | {"level":"debug","ts":1622734135.5657556,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:55 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["QmxTNDV3WkFJUWROa2tmejR1RlF5dHRGRDBzc2Nuclc"]}}
caddy_1  | {"level":"debug","ts":1622734135.8232574,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:55 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["T1RlOWw1YnNueExjUFNUeXREcUpUVG10emVtSHc1bkE"]}}
caddy_1  | {"level":"debug","ts":1622734135.824062,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:55 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["enRKU1Z5bXpuVm40ckw1bmY5Vkw3MWx4Z3N6RVZrZG8"]}}
caddy_1  | {"level":"debug","ts":1622734136.0820508,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:56 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["SkI5eGlteENCcFhVWHN3UktqbmdVWEdtaDBNN3dDSmc"]}}
caddy_1  | {"level":"debug","ts":1622734136.0827632,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:56 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["UE96U2Zuelk2YlZnMVlJZmphazNjMlpYVnk0WkdtZks"]}}
caddy_1  | {"level":"debug","ts":1622734136.3412068,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:56 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["Z3VKbk42eE9kWWV5UmlZQU5sSXJjM1JWWU90dHZOaEI"]}}
caddy_1  | {"level":"debug","ts":1622734136.342266,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:56 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["cVk4RHdXR21zMjVBMHpFTXA4Vk1wMkR4OTNNMGNyRG0"]}}
caddy_1  | {"level":"debug","ts":1622734136.601023,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:56 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["QVFzQUxaZEc3T01QQlB1REl0QW8zUGp5ZDZLZE9QdWk"]}}
caddy_1  | {"level":"debug","ts":1622734136.6018295,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:56 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["OHJURHM2QWdkWXZZdlF3alJXV2xlME1GbE9PSlJaNlY"]}}
caddy_1  | {"level":"debug","ts":1622734136.8601387,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:56 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["VDhqU1A5ZEpuNTlhSjJFdHpvbnZYc0U5TUZpZDZVQXg"]}}
caddy_1  | {"level":"debug","ts":1622734136.8608942,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:56 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["cTZMdU9EaXJ6RzVjTVQ2RVJpT2dkd1I4V0NNV201NHQ"]}}
caddy_1  | {"level":"debug","ts":1622734137.1193275,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:57 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["aGwyRExxVGdJRWVNVXp3SFJYTTR0cGZLbXdDNzNNUlg"]}}
caddy_1  | {"level":"debug","ts":1622734137.1200194,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:57 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["WWdpVlAxSVNZd3ZiUm1QY1A0UElkakVYZHB6dGxNdWo"]}}
caddy_1  | {"level":"debug","ts":1622734137.3787124,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:57 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["M3dtT1JuTTVMOTZsa1JtZkhtMTYyeFlWR1loWGNDNUI"]}}
caddy_1  | {"level":"debug","ts":1622734137.3794732,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:57 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["Wms2TEQ4ZUhvMkFER3llMjhEeVoyYnAwTnIzT243ZzE"]}}
caddy_1  | {"level":"debug","ts":1622734137.6382,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:57 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["d2k2a2QxcW45UHJkWjk1QVJsMVpYc3FKbGRHYjhiY0Q"]}}
caddy_1  | {"level":"debug","ts":1622734137.638954,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:57 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["NXpHVDR6S2YycFN6MHFjUnUxV0xkWVRlTURlN0MwWm4"]}}
caddy_1  | {"level":"debug","ts":1622734137.897929,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:57 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["TW9Pd05WYlp4QXo2YmhtMFp1ZEU2MTFKT3hrUlZSa2U"]}}
caddy_1  | {"level":"debug","ts":1622734137.8988328,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:57 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["YVRXQjlTbnQ2bTZIT3BjNlFCdERMb1V6S3pMTjFxbTY"]}}
caddy_1  | {"level":"debug","ts":1622734138.1568336,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:58 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["VU1DVXF6MU4wVTd0MEFqeldmYmFIME1USkNOWjluRUk"]}}
caddy_1  | {"level":"debug","ts":1622734138.1577253,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:58 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["WUJWTGNQWTh4T3VkRUZodmR0QngzRWFxOFRyV1pxdlI"]}}
caddy_1  | {"level":"debug","ts":1622734138.4152513,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:58 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["bzYzVzhTREZoYVdPTHEyd2VxbUtjR1hUeGVjNjZPYjU"]}}
caddy_1  | {"level":"debug","ts":1622734138.4160254,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:58 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["ajFWUFFuNTFpeVRURGpWQ3pMdlh5MHJZeDRvaUd1OFU"]}}
caddy_1  | {"level":"debug","ts":1622734138.6752775,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:58 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["c1g2ak90UVZWam9XaXIwVUFYQmZ1c3JRZDFBZTl1ZTQ"]}}
caddy_1  | {"level":"debug","ts":1622734138.6762612,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:58 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["c3hxdzA5RTRDTmhkZ3JVbXV4aHE4dVplYkVodmdSTUE"]}}
caddy_1  | {"level":"debug","ts":1622734138.9343765,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:58 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["cmJINGxwaFQ3ejl4ano2UGxGRUVWVTlOSmUwZmVmMTY"]}}
caddy_1  | {"level":"debug","ts":1622734138.935135,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:58 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["ZUpvRm1kc1RqOUpTbUpjQnR5Q2dLdzAwbWFDbG9ieGI"]}}
caddy_1  | {"level":"debug","ts":1622734139.1940045,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:59 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["MklMR0J5SjdKODRPOXFDTEpkdkNXeVZpWWxRSVdyOU0"]}}
caddy_1  | {"level":"debug","ts":1622734139.1947815,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:59 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["amNtdHJkZ3dabXlHMGpoSkRjSHIxR3BtRW5uWkJOSHc"]}}
caddy_1  | {"level":"debug","ts":1622734139.4522796,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:59 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["QWlhMGxLZXJjS2pmcHdnVDRUWUs1eWtyT2VRRkR0ZkE"]}}
caddy_1  | {"level":"debug","ts":1622734139.4530003,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["882"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:59 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/B4d7x7t6MhvwtHbmkdUqPYBRqEOgHcOB"],"Replay-Nonce":["MmVHRjA4MEhBVTQzaGxuRTBPV0I5NzlQS2Vpd1RwcU0"]}}
caddy_1  | {"level":"debug","ts":1622734139.717766,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["no-store"],"Content-Length":["874"],"Content-Type":["application/json"],"Date":["Thu, 03 Jun 2021 15:28:59 GMT"],"Link":["<https://ca.domain.local/acme/acme/directory>;rel=\"index\""],"Location":["https://ca.domain.local/acme/acme/authz/75maiCUV9eH9wxIEOvcQeGjLf75Sqkjs"],"Replay-Nonce":["azdLR0ljTVoxWGt0cTQyMWZhdEV5c2U4b1Foc05pbVo"]}}

2d. Workaround(s)

Double checking the config, i guess?

2e. Relevant links

/

3. Tutorial (minimal steps to reproduce the bug)

  1. Start step-ca

  2. create caddyfile

    {
    email caddy@domain.local
    acme_ca https://ca.domain.local/acme/acme/directory
    ocsp_stapling off      # <- step-ca doesn't support this feature
    debug
}
frontend.domain.local {
    encode zstd gzip
    reverse_proxy https://back.domain.local {
            header_up Host {http.reverse_proxy.upstream.hostport}
            header_up X-Forwarded-Host {host}
    }
}

    Just note that the used domain must be non-existent

  3. start caddy and look at the logs. Caddy should immediately start flooding your CA server. Just ~2min were over 19000 lines of log messages for this demonstration.

Hope that helps! Just let me know if you need more informations. Thank you for helping!

mholt commented 3 years ago

I think there must be an error in your setup somewhere. Your config only has the domain frontend.domain.local in it, but that does not appear anywhere else on this page or in your logs.

MCWertGaming commented 3 years ago

I have changed the domain for privacy reasons. The frontend.domain.local is basically an example for gitea.domain.local, drone.domain.local and the other ones. Please open this therefore again @mholt. Sorry for the misleading!

mholt commented 3 years ago

Ok. Please update your post to use all real domains and exact output without changes or redactions, as per the instructions, and we'll look at this again. Thanks.

DO NOT REDACT INFORMATION except for credentials. ...

  • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
  • Note that ignoring this advice may result in delays, or even in your issue being closed. disappointed Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.