crashed/stopped container results in a 308 loop (bug?)

[easy-easy]

Hello,

I am using docker compose for my setup. I have dev instances of my containers, which I'm running with

docker compose up app-dev

If this app crashes, or I stop it via ctrl-c, the container itself is not removed it get the status "stopped":

824710605cde app-dev:latest "/usr/local/bin/pyth…" 2 hours ago Exited (0) 10 minutes ago

If I do any requests into this stopped container, this result in a 308 redirect to itself:

[easy@workstation app$] curl -v 'https://app.dev.mb/.api/call/index'
* Host app.dev.mb:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.x.x
*   Trying 192.168.x.x:443...
* Connected to app.dev.mb (192.168.x.x) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=wildmb
*  start date: Sep 13 10:27:47 2023 GMT
*  expire date: Dec 16 10:27:47 2025 GMT
*  subjectAltName: host "app.dev.mb" matched cert's "*.dev.mb"
*  issuer: CN=Easy-RSA CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://app.dev.mb/.api/call/index
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: app.dev.mb]
* [HTTP/2] [1] [:path: /.api/call/index]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET /.api/call/index HTTP/2
> Host: app.dev.mb
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/2 308 
< date: Wed, 20 Dec 2023 07:53:53 GMT
< location: https://app.dev.mb/.api/call/index
< server: Caddy
< server: Caddy
< content-length: 0
< 
* Connection #0 to host app.dev.mb left intact

The reverse proxy entry for the stopped container is still in the caddy configuration. But the dial just have ":80" instead "container-ip:80", so I think this is a proxy error and not a caddy error. (But I'm not 100% sure about that).

If this is not a bug, how can I avoid this 308 redirects of crashed/stopped but not removed containers?

Reagards, easy.

[francislavoie]

What did you use as the labels? What's in Caddy's logs?

[easy-easy]

my labels are the following:

    labels:
      caddy: app.dev.mb
      caddy.0_import: common
      caddy.1_import: log app-dev
      caddy.reverse_proxy: '{{ upstreams 80 }}'

The snippets in my Caddyfile:

(common) {
    tls /etc/caddy/wildmb.crt /etc/caddy/wildmb.key
}

(log) {
    log {
        output file /var/log/caddy/{args[0]}.log {
            roll_size: 1gb
            roll_local_time
            roll_keep: 3
            roll_keep_for: 120d
        }
        format json
    }
}

The logentry for the request:

{"level":"info","ts":1703059069.9339566,"logger":"http.log.access.log6","msg":"handled request","request":{"remote_ip":"172.22.0.1","remote_port":"41494","client_ip":"172.22.0.1","proto":"HTTP/2.0","method":"GET","host":"app.dev.mb","uri":"/.api/call/index","headers":{"User-Agent":["curl/8.5.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"app.dev.mb"}},"bytes_read":0,"user_id":"","duration":0.000545562,"size":0,"status":308,"resp_headers":{"Server":["Caddy","Caddy"],"Location":["https://app.dev.mb/.api/call/index"],"Date":["Wed, 20 Dec 2023 07:57:49 GMT"],"Content-Length":["0"]}}

Caddyversion: 2.7.4

In the caddylog itself, there is no error or notice entry.

So just a basic setup with a self signed wildcard certificate instead a lets encrypt one. But there is no SSL-Issue whatsoever.

Regards.

[francislavoie]

Weird. I'll need to defer to @lucaslorentz about this; why do stopped containers still generate Caddyfile config?

[easy-easy]

I dont know.

Here is the config entry with a running container:

                  {
                     "handle" : [
                        {
                           "handler" : "subroute",
                           "routes" : [
                              {
                                 "handle" : [
                                    {
                                       "handler" : "reverse_proxy",
                                       "upstreams" : [
                                          {
                                             "dial" : "172.22.0.31:80"
                                          }
                                       ]
                                    }
                                 ]
                              }
                           ]
                        }
                     ],
                     "match" : [
                        {
                           "host" : [
                              "app.dev.mb"
                           ]
                        }
                     ],
                     "terminal" : true
                  }

and here if the container is stopped: (but not removed)

                  {
                     "handle" : [
                        {
                           "handler" : "subroute",
                           "routes" : [
                              {
                                 "handle" : [
                                    {
                                       "handler" : "reverse_proxy",
                                       "upstreams" : [
                                          {
                                             "dial" : ":80"
                                          }
                                       ]
                                    }
                                 ]
                              }
                           ]
                        }
                     ],
                     "match" : [
                        {
                           "host" : [
                              "app.dev.mb"
                           ]
                        }
                     ],
                     "terminal" : true
                  },

As you can see, it is basically the same, just the ip from the container is missing. As soon as I remove the container with docker container rm ... the config entry also disappears, what makes sense..

I think the best would be a label entry like

caddy.no_upstreams: "respond 'Error...' 500"

or something like this, so if there a no upstreams available, that the config entry is a direct response entry.

Regards...

(edit: changed example label entry it match respond config of caddy)

[lucaslorentz]

Weird. I'll need to defer to @lucaslorentz about this; why do stopped containers still generate Caddyfile config?

It shouldn't scan stopped containers unless --scan-stopped-containers is set.

But it looks like I accidentally made scan-stopped-containers default to true. I will fix that in another PR. A breaking change, unfortunately, but changing to scan stopped containers by default was an even bigger breaking change, so it's better to restore it to false.