Formatting looks OK but getting "Invalid request headers"

[spikespaz]

Hey there,

A little while ago my Vaultwarden instance stopped working. I checked the web frontend and the interface was down with (first a CloudFlare host error, and playing with it made it change into) an invalid SSL certificate.

Here are the logs, truncated for brevity:

caddy          | {"level":"info","ts":1654723317.160815,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"linode-vaultwarden.spikespaz.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy          | {"level":"error","ts":1654723317.2791078,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"linode-vaultwarden.spikespaz.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for linode-vaultwarden.spikespaz.com (probably OK if presenting failed)"}
caddy          | {"level":"error","ts":1654723317.36962,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"linode-vaultwarden.spikespaz.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[linode-vaultwarden.spikespaz.com] solving challenges: presenting for challenge: adding temporary record for zone spikespaz.com.: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme-v02.api.letsencrypt.org/acme/order/411258510/96010725016) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy          | {"level":"info","ts":1654723317.372098,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["linode-vaultwarden.spikespaz.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"spikespaz@outlook.com"}
caddy          | {"level":"info","ts":1654723317.3731222,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["linode-vaultwarden.spikespaz.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"spikespaz@outlook.com"}
caddy          | {"level":"info","ts":1654723325.842115,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"linode-vaultwarden.spikespaz.com","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
caddy          | {"level":"error","ts":1654723325.9414904,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"linode-vaultwarden.spikespaz.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for linode-vaultwarden.spikespaz.com (probably OK if presenting failed)"}
caddy          | {"level":"error","ts":1654723329.863269,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"linode-vaultwarden.spikespaz.com","issuer":"acme.zerossl.com-v2-DV90","error":"[linode-vaultwarden.spikespaz.com] solving challenges: presenting for challenge: adding temporary record for zone spikespaz.com.: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/V9_HdrdzKjVUHDrqGQw_ew) (ca=https://acme.zerossl.com/v2/DV90)"}
caddy          | {"level":"error","ts":1654723329.8633559,"logger":"tls.obtain","msg":"will retry","error":"[linode-vaultwarden.spikespaz.com] Obtain: [linode-vaultwarden.spikespaz.com] solving challenges: presenting for challenge: adding temporary record for zone spikespaz.com.: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/V9_HdrdzKjVUHDrqGQw_ew) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":13.350819347,"max_duration":2592000}
caddy          | {"level":"info","ts":1654723390.2530217,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"linode-vaultwarden.spikespaz.com","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy          | {"level":"error","ts":1654723390.3825397,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"linode-vaultwarden.spikespaz.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for linode-vaultwarden.spikespaz.com (probably OK if presenting failed)"}
caddy          | {"level":"error","ts":1654723390.4436834,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"linode-vaultwarden.spikespaz.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[linode-vaultwarden.spikespaz.com] solving challenges: presenting for challenge: adding temporary record for zone spikespaz.com.: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/51143733/2796180054) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
caddy          | {"level":"info","ts":1654723397.8965223,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"linode-vaultwarden.spikespaz.com","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
caddy          | {"level":"error","ts":1654723398.0058784,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"linode-vaultwarden.spikespaz.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for linode-vaultwarden.spikespaz.com (probably OK if presenting failed)"}
caddy          | {"level":"error","ts":1654723400.164129,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"linode-vaultwarden.spikespaz.com","issuer":"acme.zerossl.com-v2-DV90","error":"[linode-vaultwarden.spikespaz.com] solving challenges: presenting for challenge: adding temporary record for zone spikespaz.com.: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/-n9DujCE-i07QAZ-deyQlg) (ca=https://acme.zerossl.com/v2/DV90)"}
caddy          | {"level":"error","ts":1654723400.1642082,"logger":"tls.obtain","msg":"will retry","error":"[linode-vaultwarden.spikespaz.com] Obtain: [linode-vaultwarden.spikespaz.com] solving challenges: presenting for challenge: adding temporary record for zone spikespaz.com.: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/-n9DujCE-i07QAZ-deyQlg) (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":83.65167139,"max_duration":2592000}

The Caddyfile:

{$DOMAIN}:443 {
  log {
    level INFO
    output file {$LOG_FILE} {
      roll_size 10MB
      roll_keep 10
    }
  }

  tls {$EMAIL} {
      dns cloudflare {$CLOUDFLARE_API_TOKEN}
  }

  # This setting may have compatibility issues with some browsers
  # (e.g., attachment downloading on Firefox). Try disabling this
  # if you encounter issues.
  encode gzip

  # Uncomment to improve security (WARNING: only use if you understand the implications!)
  header {
       # Enable HTTP Strict Transport Security (HSTS)
       Strict-Transport-Security "max-age=31536000;"
       # Enable cross-site filter (XSS) and tell browser to block detected attacks
       X-XSS-Protection "1; mode=block"
       # Disallow the site to be rendered within a frame (clickjacking protection)
       X-Frame-Options "DENY"
       # Prevent search engines from indexing (optional)
       X-Robots-Tag "none"
       # Server name removing
       -Server
  }

  # Notifications redirected to the WebSocket server
  reverse_proxy /notifications/hub vaultwarden:3012

  # Proxy everything else to Rocket
  reverse_proxy vaultwarden:80 {
       # Send the true remote IP to Rocket, so that vaultwarden can put this in the
       # log, so that fail2ban can ban the correct IP.
       header_up X-Real-IP {remote_host}
  }
}

Lastly the docker-compose.yaml:

version: '3'

services:
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: always
    environment:
      - DOMAIN=https://linode-vaultwarden.spikespaz.com
      - WEBSOCKET_ENABLED=true
      - SIGNUPS_ALLOWED=false
      - ADMIN_TOKEN=password1234!
      - LOG_FILE=/data/vaultwarden.log
    volumes:
      - /home/adminuser/config/vaultwarden:/data

  caddy:
    image: spikespaz/caddy-with-cloudflare:latest-alpine
    container_name: caddy
    restart: always
    ports:
      - 8080:8080
      - 443:443
    volumes:
      - /home/adminuser/config/caddy/Caddyfile:/etc/caddy/Caddyfile:ro
      - /home/adminuser/config/caddy/config:/config
      - /home/adminuser/config/caddy/data:/data
    environment:
      - DOMAIN=https://linode-vaultwarden.spikespaz.com
      - CLOUDFLARE_API_TOKEN=v1.0-1914e277-e85c-4020-9452-692e114e6446
      - EMAIL=spikespaz@outlook.com
      - LOG_FILE=/data/access.log

  fail2ban:
    image: crazymax/fail2ban:latest
    container_name: fail2ban
    restart: always
    network_mode: 'host'
    privileged: true
    cap_add:
      - NET_ADMIN
      - NET_RAW
    volumes:
      - /home/adminuser/config/fail2ban:/data
      - /home/adminuser/config/vaultwarden:/vaultwarden:ro
      - /var/log:/var/log:ro
    environment:
      - F2B_LOG_TARGET=/data/fail2ban.log
      - F2B_DB_PURGE_AGE=60d
      - F2B_IPTABLES_CHAIN=INPUT

  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WATCHTOWER_CLEANUP=true

[mholt]

That (misleading) error from Cloudflare means your credentials are wrong; see #27, and probably a duplicate of #38.

[spikespaz]

I saw the thing in the readme. I tried creating a fresh token, same way as I did originally, by creating a CA API key under my account. It didn't work, and this just happened suddenly. Does my handling of the environment variable look correct?

[mholt]

Hmm. I think so, but:

CLOUDFLARE_API_TOKEN=v1.0-1914e277-e85c-4020-9452-692e114e6446

I don't think I've ever seen my own tokens have v1.0- prefixed; also, I'm assuming you've obfuscated your actual token right?

(I'm also not super familiar with Docker Compose so I can't verify that for you.)

Running caddy environ or caddy run --environ will show you the environment Caddy sees, and should verify for you whether your env is set up properly.

[spikespaz]

Yeah, that's a fake token. The real one is longer, but it does have the prefix there. And it was working for months, until now.

[mholt]

I don't really know what to tell ya then. Caddy code hasn't changed, the plugin hasn't changed, if it just suddenly stopped and your own network/system/infrastructure didn't change, then something with Cloudflare must have changed. I would ask Cloudflare why your auth token suddenly stopped working.