Caddy frequently failing to start correctly on Windows containers

[NickMoores]

Hello!

I've been trying out the Windows containers recently but have come a bit stuck. I'm sure I must be doing something incorrectly but I can't determine what. My Caddy file resembles this - have omitted the real domains:

:80, :443 {
    redir https://{$HOSTNAME}{uri}
}

{$HOSTNAME} {
    reverse_proxy https://a.b.c/ {
        header_up Host a.b.c:443
    }
}

I launch the container using the following:

docker run -p 80:80 -p 443:443 -p 443:443/udp -e HOSTNAME=x.y.local  -v ${PWD}/:c:/etc/caddy/ caddy:2-windowsservercore-ltsc2022

More often than not (approx 80-90% of the time), Caddy doesn't appear to bind to ports 80 nor 443 inside the container. I confirm this by opening a cmdline in the container and running netstat which shows nothing listening on port 80 nor 443.

Here is the log output when this happens:

{"level":"info","ts":1692110395.8251982,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1692110395.83172,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1692110395.8322265,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1692110395.8322265,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1692110395.8322265,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"info","ts":1692110395.8322265,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00047ac80"}
{"level":"info","ts":1692110395.850229,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:c:\\data\\caddy"}
{"level":"info","ts":1692110395.850229,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":1692110395.9818804,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1692110395.9850142,"msg":"note: NSS support is not available on your platform"}
{"level":"info","ts":1692110395.9850142,"msg":"define JAVA_HOME environment variable to use the Java trust"}

When it works, it outputs the following:

{"level":"info","ts":1692110187.6819425,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1692110187.716456,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1692110187.716456,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1692110187.716456,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1692110187.716456,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"info","ts":1692110187.717458,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00043e000"}
{"level":"info","ts":1692110187.72074,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1692110187.72074,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1692110187.72074,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1692110187.72074,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["x.y.local"]}
{"level":"info","ts":1692110187.7352862,"logger":"tls.obtain","msg":"acquiring lock","identifier":"x.y.local"}
{"level":"info","ts":1692110187.7352862,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:c:\\data\\caddy"}
{"level":"info","ts":1692110187.7352862,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1692110187.7406073,"logger":"tls.obtain","msg":"lock acquired","identifier":"x.y.local"}
{"level":"info","ts":1692110187.7406073,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"x.y.local"}
{"level":"info","ts":1692110187.7406073,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"x.y.local"}
{"level":"info","ts":1692110187.7406073,"logger":"tls.obtain","msg":"releasing lock","identifier":"x.y.local"}
{"level":"warn","ts":1692110187.7406073,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [x.y.local]: no OCSP server specified in certificate","identifiers":["x.y.local"]}
{"level":"warn","ts":1692110187.855247,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1692110187.8569531,"msg":"note: NSS support is not available on your platform"}
{"level":"info","ts":1692110187.8569531,"msg":"define JAVA_HOME environment variable to use the Java trust"}

I haven't yet been able to identify a pattern as to why it's not working, but I think the problem is limited to the Windows containers because when using the equivilent Linux container I cannot reproduce the issue, even with the same Caddyfile.

Is something going on here that I'm not quite grasping? My understanding is that the behaviour of Caddy, with regards to certificates, is different here because I am using a ".local" TLD, and I think this could be a contributing factor but should still work, as far as I can tell.

Thanks for all your hard work in making these images available!

[francislavoie]

Your config is kinda strange:

:80, :443 {
  redir https://{$HOSTNAME}{uri}
}

{$HOSTNAME} {
  reverse_proxy https://a.b.c/ {
      header_up Host a.b.c:443
  }
}

:80, :443 doesn't really make sense because you don't have a cert configured for :443 to do anything. I suggest you remove that. Caddy sets up HTTP->HTTPS redirects anyway, so I don't think you need :80 either. I think that site block might be breaking things.

Also reverse_proxy https://a.b.c/ is invalid, because upstream addresses cannot have a path (i.e. the trailing /). Caddy would fail to start with that, so I assume you don't have that normally -- please don't obfuscate your config, it only makes it harder for us to help. Domains are public information, they're not secret.

Also you can use header_up Host {upstream_hostport} which lets you avoid repeating the upstream address a second time.

Please also enable the debug global option and try again. Make an example request with curl -v and show the output.

[NickMoores]

Thanks for the pointers - really appreciate it. I've tried the suggestions but without any luck. Apologies for the obfuscation - our cyber security team are really anxious about us posting URLs for internal tools (even when they're on public domains!) - so I've tried to produce a more minimal example with recognised domains. The below is exactly what I'm using now, though.

Caddyfile:

{
    debug
}

{$HOSTNAME} {
    reverse_proxy https://google.co.uk {
        header_up Host {upstream_hostport}
    }
}

Run command:

docker run -p 80:80 -p 443:443 -p 443:443/udp -e HOSTNAME=e105109.local  -v ${PWD}/:c:/etc/caddy/ caddy:2-windowsservercore-ltsc2022

Log output:

{"level":"info","ts":1692177578.749619,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1692177578.760149,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1692177578.7613542,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1692177578.7613542,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1692177578.76267,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000553300"}
{"level":"debug","ts":1692177578.7764177,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["e105109.local"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.reverse_proxy.upstream.hostport}"]}}},"transport":{"protocol":"http","tls":{}},"upstreams":[{"dial":"google.co.uk:443"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
{"level":"info","ts":1692177578.781144,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:c:\\data\\caddy"}
{"level":"info","ts":1692177578.781144,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":1692177578.9143736,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1692177578.9178104,"msg":"note: NSS support is not available on your platform"}
{"level":"info","ts":1692177578.9178104,"msg":"define JAVA_HOME environment variable to use the Java trust"}

curl -v output:

curl -v http://localhost/
*   Trying 127.0.0.1:80...
*   Trying [::1]:80...
* connect to ::1 port 80 failed: Connection refused
* connect to 127.0.0.1 port 80 failed: Timed out
* Failed to connect to localhost port 80 after 21049 ms: Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to localhost port 80 after 21049 ms: Couldn't connect to server

Output from netstat -a ran from inside the container:

C:\>netstat -a

Active Connections

  Proto  Local Address          Foreign Address        State
  TCP    0.0.0.0:135            24974e22212e:0         LISTENING
  TCP    0.0.0.0:5985           24974e22212e:0         LISTENING
  TCP    0.0.0.0:47001          24974e22212e:0         LISTENING
  TCP    0.0.0.0:49152          24974e22212e:0         LISTENING
  TCP    0.0.0.0:49153          24974e22212e:0         LISTENING
  TCP    0.0.0.0:49154          24974e22212e:0         LISTENING
  TCP    0.0.0.0:49155          24974e22212e:0         LISTENING
  TCP    0.0.0.0:49156          24974e22212e:0         LISTENING
  TCP    127.0.0.1:2019         24974e22212e:0         LISTENING
  TCP    [::]:135               24974e22212e:0         LISTENING
  TCP    [::]:5985              24974e22212e:0         LISTENING
  TCP    [::]:47001             24974e22212e:0         LISTENING
  TCP    [::]:49152             24974e22212e:0         LISTENING
  TCP    [::]:49153             24974e22212e:0         LISTENING
  TCP    [::]:49154             24974e22212e:0         LISTENING
  TCP    [::]:49155             24974e22212e:0         LISTENING
  TCP    [::]:49156             24974e22212e:0         LISTENING
  UDP    0.0.0.0:5353           *:*
  UDP    0.0.0.0:5355           *:*
  UDP    127.0.0.1:64121        127.0.0.1:64121        
  UDP    [::]:5353              *:*
  UDP    [::]:5355              *:*

If I switch to Linux containers, the situation improves:

Run command:

docker run -p 80:80 -p 443:443 -p 443:443/udp -e HOSTNAME=e105109.local  -v ${PWD}/Caddyfile:/etc/caddy/Caddyfile caddy:2

Curl -v output:

curl -v http://localhost/
*   Trying 127.0.0.1:80...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://localhost/
< Server: Caddy
< Date: Wed, 16 Aug 2023 12:01:01 GMT
< Content-Length: 0
<
* Closing connection 0

[francislavoie]

Do earlier versions of Caddy (e.g. 2.6.4) work any better?

Frankly I don't know anyone else who uses these Windows containers. I don't understand the usecase for them either 🤷‍♂️ I think we mostly just offer them for completions sake.