search

lucaslorentz / caddy-docker-proxy

Caddy as a reverse proxy for Docker
MIT License
2.86k stars 168 forks source link

Caddy is not reading my labels #547

Closed sowinski closed 10 months ago

sowinski commented 10 months ago

Hi,

this is my docker-compose for caddy:

version: "3.7"
services:
  caddy:
    image: lucaslorentz/caddy-docker-proxy:2.8.9-alpine
    ports:
      - 80:80
      - 443:443
    environment:
      - CADDY_INGRESS_NETWORKS=caddy
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy_data:/data
    restart: unless-stopped

networks:
  caddy:
    external: true

volumes:
  caddy_data: {}

And this is my docker-compose for my django project.

version: '3.8'
services:
  django:
    image: django-backend:latest
    ports:
      - "8000:8000"
    command: gunicorn myapp.wsgi --bind 0.0.0.0:8000
    env_file:
      - ../django.env
    environment:
      CELERY_BROKER_URL: ${CELERY_BROKER_URL}
    volumes:
      - logvolume:/code/logs
    networks:
      - caddy
    labels:
      caddy: www.example.com
      caddy.reverse_proxy: "{{upstreams 8000}}"

volumes:
  logvolume:

networks:
  caddy:
    external: true

The first time i used the port 9999 and changed it later to 8000.

I restarted several times caddy and also my django project. I can locally connect to the django project and it works on port 8000. If I connect externally from www.example.com i can see in the caddy logs that caddy is still trying to forward it to port 9999.

I do not undestand why it is not updating.

Do I have to implement a healthcheck or something else to make caddy update this to port 8000?

francislavoie commented 10 months ago

Please show your logs.

sowinski commented 10 months ago 
sudo docker-compose up --build
[+] Running 1/0
 ✔ Container caddy_caddy_1  Created                                                                                                                                                                 0.0s 
Attaching to caddy_1
caddy_1  | {"level":"info","ts":1700776180.582202,"logger":"docker-proxy","msg":"Running caddy proxy server"}
caddy_1  | {"level":"info","ts":1700776180.5840368,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy_1  | {"level":"info","ts":1700776180.5844781,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1  | {"level":"info","ts":1700776180.5844936,"logger":"docker-proxy","msg":"Running caddy proxy controller"}
caddy_1  | {"level":"info","ts":1700776180.5854888,"logger":"docker-proxy","msg":"Start","CaddyfilePath":"","LabelPrefix":"caddy","PollingInterval":30,"ProxyServiceTasks":true,"ProcessCaddyfile":true,"ScanStoppedContainers":true,"IngressNetworks":"[caddy]","DockerSockets":[""],"DockerCertsPath":[""],"DockerAPIsVersion":[""]}
caddy_1  | {"level":"info","ts":1700776180.5866935,"logger":"docker-proxy","msg":"Connecting to docker events","DockerSocket":""}
caddy_1  | {"level":"info","ts":1700776180.588175,"logger":"docker-proxy","msg":"IngressNetworksMap","ingres":"map[a48a2cad8e93313766a61789781a28adc75c571c86cd47b4e09aa713d1c3ae9c:true caddy:true]"}
caddy_1  | {"level":"info","ts":1700776180.595984,"logger":"docker-proxy","msg":"Swarm is available","new":false}
caddy_1  | {"level":"info","ts":1700776180.60986,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy 172.29.0.3:8000 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
caddy_1  | {"level":"info","ts":1700776180.6102848,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"www.example.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.29.0.3:8000\"},{\"dial\":\"172.29.0.4:9999\"},{\"dial\":\":9999\"},{\"dial\":\":9999\"},{\"dial\":\":80\"}]}]}]}],\"terminal\":true}]}}}}}"}
caddy_1  | {"level":"info","ts":1700776180.6103382,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
caddy_1  | {"level":"info","ts":1700776180.6109803,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"42482","headers":{"Accept-Encoding":["gzip"],"Content-Length":["367"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy_1  | {"level":"info","ts":1700776180.611386,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
caddy_1  | {"level":"info","ts":1700776180.6114962,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1  | {"level":"info","ts":1700776180.6115198,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1  | {"level":"info","ts":1700776180.611741,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1  | {"level":"info","ts":1700776180.6118507,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
caddy_1  | {"level":"info","ts":1700776180.611999,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1  | {"level":"info","ts":1700776180.612044,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1  | {"level":"info","ts":1700776180.6120574,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.example.com"]}
caddy_1  | {"level":"info","ts":1700776180.6131058,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1  | {"level":"info","ts":1700776180.6131315,"logger":"admin.api","msg":"load complete"}
caddy_1  | {"level":"info","ts":1700776180.6133792,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy_1  | {"level":"info","ts":1700776180.613424,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0006bc600"}
caddy_1  | {"level":"info","ts":1700776180.6136477,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy_1  | {"level":"info","ts":1700776180.614009,"logger":"tls","msg":"finished cleaning storage units"}
caddy_1  | {"level":"info","ts":1700776180.6143267,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
caddy_1  | {"level":"info","ts":1700776189.0448728,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy :8000 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
caddy_1  | {"level":"info","ts":1700776189.045548,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"www.example.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\":8000\"},{\"dial\":\"172.29.0.4:9999\"},{\"dial\":\":9999\"},{\"dial\":\":9999\"},{\"dial\":\":80\"}]}]}]}],\"terminal\":true}]}}}}}"}
caddy_1  | {"level":"info","ts":1700776189.0455964,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
caddy_1  | {"level":"info","ts":1700776189.0471437,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"36668","headers":{"Accept-Encoding":["gzip"],"Content-Length":["357"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy_1  | {"level":"info","ts":1700776189.04832,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
caddy_1  | {"level":"info","ts":1700776189.048497,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1  | {"level":"info","ts":1700776189.0485358,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1  | {"level":"info","ts":1700776189.0490217,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1  | {"level":"info","ts":1700776189.0491111,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1  | {"level":"info","ts":1700776189.0492048,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1  | {"level":"info","ts":1700776189.049262,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.example.com"]}
caddy_1  | {"level":"info","ts":1700776189.0492947,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy_1  | {"level":"info","ts":1700776189.0501091,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1  | {"level":"info","ts":1700776189.05016,"logger":"admin.api","msg":"load complete"}
caddy_1  | {"level":"info","ts":1700776189.050379,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
caddy_1  | {"level":"info","ts":1700776189.0505602,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy_1  | {"level":"info","ts":1700776192.842112,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy 172.29.0.3:8000 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
caddy_1  | {"level":"info","ts":1700776192.8425071,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"www.example.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.29.0.3:8000\"},{\"dial\":\"172.29.0.4:9999\"},{\"dial\":\":9999\"},{\"dial\":\":9999\"},{\"dial\":\":80\"}]}]}]}],\"terminal\":true}]}}}}}"}
caddy_1  | {"level":"info","ts":1700776192.8425362,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
caddy_1  | {"level":"info","ts":1700776192.8434806,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"36680","headers":{"Accept-Encoding":["gzip"],"Content-Length":["367"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy_1  | {"level":"info","ts":1700776192.8449876,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
caddy_1  | {"level":"info","ts":1700776192.8451831,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1  | {"level":"info","ts":1700776192.845202,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1  | {"level":"info","ts":1700776192.8456044,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1  | {"level":"info","ts":1700776192.8456333,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1  | {"level":"info","ts":1700776192.8456771,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1  | {"level":"info","ts":1700776192.8456895,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.example.com"]}
caddy_1  | {"level":"info","ts":1700776192.8457038,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy_1  | {"level":"info","ts":1700776192.846989,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1  | {"level":"info","ts":1700776192.847001,"logger":"admin.api","msg":"load complete"}
caddy_1  | {"level":"info","ts":1700776192.847502,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy_1  | {"level":"info","ts":1700776192.8476045,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
caddy_1  | {"level":"error","ts":1700776220.799892,"logger":"http.log.error","msg":"dial tcp :9999: connect: connection refused","request":{"remote_ip":"172.29.0.1","remote_port":"54236","client_ip":"172.29.0.1","proto":"HTTP/2.0","method":"GET","host":"www.example.com","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.115 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\""],"Sec-Ch-Ua-Platform":["\"Linux\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Cache-Control":["max-age=0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.example.com"}},"duration":0.000555354,"status":502,"err_id":"ebd259cur","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}
caddy_1  | {"level":"info","ts":1700776305.57138,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy :8000 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
caddy_1  | {"level":"info","ts":1700776305.5729163,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"www.example.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\":8000\"},{\"dial\":\"172.29.0.4:9999\"},{\"dial\":\":9999\"},{\"dial\":\":9999\"},{\"dial\":\":80\"}]}]}]}],\"terminal\":true}]}}}}}"}
caddy_1  | {"level":"info","ts":1700776305.5729902,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
caddy_1  | {"level":"info","ts":1700776305.5753102,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"34442","headers":{"Accept-Encoding":["gzip"],"Content-Length":["357"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy_1  | {"level":"info","ts":1700776305.576389,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy_1  | {"level":"info","ts":1700776305.5765822,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1  | {"level":"info","ts":1700776305.5766027,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1  | {"level":"info","ts":1700776305.577017,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1  | {"level":"info","ts":1700776305.5770495,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1  | {"level":"info","ts":1700776305.577086,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1  | {"level":"info","ts":1700776305.577091,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.example.com"]}
caddy_1  | {"level":"info","ts":1700776305.5771034,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy_1  | {"level":"info","ts":1700776305.577657,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1  | {"level":"info","ts":1700776305.5776682,"logger":"admin.api","msg":"load complete"}
caddy_1  | {"level":"info","ts":1700776305.577816,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
caddy_1  | {"level":"info","ts":1700776305.5779428,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy_1  | {"level":"info","ts":1700776309.1495879,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
caddy_1  | {"level":"info","ts":1700776309.149949,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"www.example.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.29.0.4:9999\"},{\"dial\":\":9999\"},{\"dial\":\":9999\"},{\"dial\":\":80\"}]}]}]}],\"terminal\":true}]}}}}}"}
caddy_1  | {"level":"info","ts":1700776309.149974,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
caddy_1  | {"level":"info","ts":1700776309.1504982,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"49218","headers":{"Accept-Encoding":["gzip"],"Content-Length":["340"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy_1  | {"level":"info","ts":1700776309.1510074,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy_1  | {"level":"info","ts":1700776309.1511235,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1  | {"level":"info","ts":1700776309.1511562,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1  | {"level":"info","ts":1700776309.1513488,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1  | {"level":"info","ts":1700776309.151398,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1  | {"level":"info","ts":1700776309.151426,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1  | {"level":"info","ts":1700776309.151437,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.example.com"]}
caddy_1  | {"level":"info","ts":1700776309.1514816,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy_1  | {"level":"info","ts":1700776309.1516933,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1  | {"level":"info","ts":1700776309.1517143,"logger":"admin.api","msg":"load complete"}
caddy_1  | {"level":"info","ts":1700776309.1519494,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
caddy_1  | {"level":"info","ts":1700776309.1529338,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}

In the logs you can still see port 9999

francislavoie commented 10 months ago 
caddy_1  | {"level":"info","ts":1700776309.1495879,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy 172.29.0.4:9999 :9999 :9999 :80\n}\n"}

This tells me that you must still have some containers that were stopped but not removed from, or some other dangling state of somekind.

Please shut down all containers using docker compose down for each project, run docker system prune -f, then bring your containers back up.

sowinski commented 10 months ago

You are right. Thank you!