search

dokku / dokku-letsencrypt

Automatic Let's Encrypt TLS Certificate installation for dokku
https://blog.semicolonsoftware.de/securing-dokku-with-lets-encrypt-tls-certificates/
MIT License
1.09k stars 93 forks source link

After upgrade, letsencrypt does not renew certificates automatically #334

Closed BananaAcid closed 1 day ago

BananaAcid commented 1 day ago

Description of problem

Upgraded to dokku version 0.34.6, 3 month ago, no configured apps have renewed certificates.

Manually triggering dokku letsencrypt:renew <app> works just fine.

Steps to reproduce

Upgrading to dokku version 0.34.6.

Is there some separate service or somthing I can check if it runs or alike?

dokku report $APP_NAME

``` -----> uname: Linux p230624-linux1 5.15.0-122-generic dokku/dokku#132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux -----> memory: total used free shared buff/cache available Mem: 11956 3889 2321 24 5745 7731 Swap: 4095 2 4093 -----> disk utilization: Filesystem Size Used Avail Use% Mounted on /dev/sda2 98G 48G 46G 52% / -----> disk inode utilization: Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 6.3M 1.6M 4.8M 25% / -----> docker version: Client: Docker Engine - Community Version: 27.0.3 API version: 1.46 Go version: go1.21.11 Git commit: 7d4bcd8 Built: Sat Jun 29 00:02:33 2024 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 27.0.3 API version: 1.46 (minimum version 1.24) Go version: go1.21.11 Git commit: 662f78c Built: Sat Jun 29 00:02:33 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.18 GitCommit: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e runc: Version: 1.7.18 GitCommit: v1.1.13-0-g58aa920 docker-init: Version: 0.19.0 GitCommit: de40ad0 -----> docker daemon info: Client: Docker Engine - Community Version: 27.0.3 Context: default Debug Mode: true Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.15.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.28.1 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 15 Running: 15 Paused: 0 Stopped: 0 Images: 81 Server Version: 27.0.3 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e runc version: v1.1.13-0-g58aa920 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 11.68GiB Name: p230624-linux1 ID: 87879cfb-0b94-472c-b240-593105bae480 Docker Root Dir: /var/lib/docker Debug Mode: false File Descriptors: 116 Goroutines: 104 System Time: 2024-10-09T10:04:32.364870862Z EventsListeners: 1 Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false -----> herokuish version: herokuish: v0.9.2 buildpacks: heroku-buildpack-multi v1.2.0 heroku-buildpack-ruby v272 heroku-buildpack-nodejs v254 heroku-buildpack-clojure v91 heroku-buildpack-python v252 heroku-buildpack-java v73 heroku-buildpack-gradle v39 heroku-buildpack-scala v98 heroku-buildpack-play v26 heroku-buildpack-php v253 heroku-buildpack-go v192 heroku-buildpack-nginx v25 buildpack-null v3 -----> dokku version: dokku version 0.34.6 -----> dokku-event-listener version: v0.17.0 -----> dokku-update version: dokku-update v0.9.4 -----> docker-container-healthchecker version: v0.11.0 -----> docker-image-labeler version: v0.8.0 -----> git version: git version 2.34.1 -----> lambda-builder version: v0.8.0 -----> netrc version: v0.10.0 ! pack binary is not available -----> plugn version: plugn: v0.16.0 -----> sigil version: v0.11.0 -----> sshcommand version: sshcommand v0.18.0 -----> dokku plugins: 00_dokku-standard 0.34.6 enabled dokku core standard plugin 20_events 0.34.6 enabled dokku core events logging plugin app-json 0.34.6 enabled dokku core app-json plugin apps 0.34.6 enabled dokku core apps plugin builder 0.34.6 enabled dokku core builder plugin builder-dockerfile 0.34.6 enabled dokku core builder-dockerfile plugin builder-herokuish 0.34.6 enabled dokku core builder-herokuish plugin builder-lambda 0.34.6 enabled dokku core builder-lambda plugin builder-nixpacks 0.34.6 enabled dokku core builder-nixpacks plugin builder-null 0.34.6 enabled dokku core builder-null plugin builder-pack 0.34.6 enabled dokku core builder-pack plugin buildpacks 0.34.6 enabled dokku core buildpacks plugin caddy-vhosts 0.34.6 enabled dokku core caddy-vhosts plugin certs 0.34.6 enabled dokku core certificate management plugin checks 0.34.6 enabled dokku core checks plugin common 0.34.6 enabled dokku core common plugin config 0.34.6 enabled dokku core config plugin couchdb 1.35.0 enabled dokku couchdb service plugin cron 0.34.6 enabled dokku core cron plugin cron-restart 0.2.0 enabled dokku cron-restart plugin docker-options 0.34.6 enabled dokku core docker-options plugin domains 0.34.6 enabled dokku core domains plugin enter 0.34.6 enabled dokku core enter plugin git 0.34.6 enabled dokku core git plugin haproxy-vhosts 0.34.6 enabled dokku core haproxy-vhosts plugin http-auth 0.10.0 enabled HTTP authentication for apps letsencrypt 0.20.3 enabled Automated installation of let's encrypt TLS certificates logs 0.34.6 enabled dokku core logs plugin mariadb 1.37.0 enabled dokku mariadb service plugin network 0.34.6 enabled dokku core network plugin nginx-cache 0.3.0 enabled Manage the app cache in NGNINX nginx-vhosts 0.34.6 enabled dokku core nginx-vhosts plugin openresty-vhosts 0.34.6 enabled dokku core openresty-vhosts plugin plugin 0.34.6 enabled dokku core plugin plugin ports 0.34.6 enabled dokku core ports plugin proxy 0.34.6 enabled dokku core proxy plugin ps 0.34.6 enabled dokku core ps plugin redirect 0.9.1 enabled Plugin for managing application redirects redis 1.37.1 enabled dokku redis service plugin registry 0.34.6 enabled dokku core registry plugin repo 0.34.6 enabled dokku core repo plugin resource 0.34.6 enabled dokku core resource plugin run 0.34.6 enabled dokku core run plugin scheduler 0.34.6 enabled dokku core scheduler plugin scheduler-docker-local 0.34.6 enabled dokku core scheduler-docker-local plugin scheduler-k3s 0.34.6 enabled dokku core scheduler-k3s plugin scheduler-null 0.34.6 enabled dokku core scheduler-null plugin shell 0.34.6 enabled dokku core shell plugin ssh-keys 0.34.6 enabled dokku core ssh-keys plugin storage 0.34.6 enabled dokku core storage plugin trace 0.34.6 enabled dokku core trace plugin traefik-vhosts 0.34.6 enabled dokku core traefik-vhosts plugin =====> texxolut app-json information App json computed selected: app.json App json global selected: app.json App json selected: =====> texxolut app information App created at: 1721907862 App deploy source: git-push App deploy source metadata: ae01d87a4e45f46ea2ae130a2d728cccc7a1ba9e App dir: /home/dokku/texxolut App locked: false =====> texxolut builder information Builder build dir: Builder computed build dir: Builder computed selected: Builder global build dir: Builder global selected: Builder selected: =====> texxolut builder-dockerfile information Builder dockerfile computed dockerfile path: Dockerfile Builder dockerfile global dockerfile path: Dockerfile Builder dockerfile dockerfile path: =====> texxolut builder-herokuish information Builder herokuish computed allowed: true Builder herokuish global allowed: true Builder herokuish allowed: =====> texxolut builder-lambda information Builder lambda computed lambdayml path: lambda.yml Builder lambda global lambdayml path: lambda.yml Builder lambda lambdayml path: =====> texxolut builder-nixpacks information Builder nixpacks computed nixpackstoml path: nixpacks.toml Builder nixpacks global nixpackstoml path: nixpacks.toml Builder nixpacks nixpackstoml path: Builder nixpacks computed no cache: false Builder nixpacks global no cache: false Builder nixpacks no cache: =====> texxolut builder-pack information Builder pack computed projecttoml path: project.toml Builder pack global projecttoml path: project.toml Builder pack projecttoml path: =====> texxolut buildpacks information Buildpacks computed stack: gliderlabs/herokuish:latest-22 Buildpacks global stack: Buildpacks list: Buildpacks stack: =====> texxolut caddy information Caddy image: lucaslorentz/caddy-docker-proxy:2.9 Caddy letsencrypt email: Caddy letsencrypt server: https://acme-v02.api.letsencrypt.org/directory Caddy log level: ERROR Caddy polling interval: 5s Caddy tls internal: false =====> texxolut ssl information Ssl dir: /home/dokku/texxolut/tls Ssl enabled: true Ssl hostnames: ancobu.com ancobu.de texxolut.com texxolut.de texxolut.hosting.texxolut.net texxolut.net texxolut.org www.ancobu.com www.ancobu.de www.texxolut.com www.texxolut.de www.texxolut.net www.texxolut.org Ssl expires at: Jan 6 08:50:14 2025 GMT Ssl issuer: C = US, O = Let's Encrypt, CN = E6 Ssl starts at: Oct 8 08:50:15 2024 GMT Ssl subject: subject=CN = texxolut.hosting.texxolut.net Ssl verified: verified by a certificate authority =====> texxolut checks information Checks disabled list: none Checks skipped list: none Checks computed wait to retire: 60 Checks global wait to retire: 60 Checks wait to retire: =====> texxolut cron-restart information Cron restart global schedule: Cron restart schedule: =====> texxolut docker options information Docker options build: Docker options deploy: --restart=on-failure:10 Docker options run: =====> texxolut domains information Domains app enabled: true Domains app vhosts: texxolut.hosting.texxolut.net texxolut.de texxolut.net texxolut.com texxolut.org www.texxolut.de www.texxolut.net www.texxolut.com www.texxolut.org ancobu.de ancobu.com www.ancobu.de www.ancobu.com Domains global enabled: true Domains global vhosts: hosting.texxolut.net =====> texxolut git information Git deploy branch: master Git global deploy branch: master Git keep git dir: false Git rev env var: GIT_REV Git sha: ae01d87a4e45f46ea2ae130a2d728cccc7a1ba9e Git source image: Git last updated at: 1721981487 =====> texxolut haproxy information Haproxy image: byjg/easy-haproxy:4.4.0 Haproxy letsencrypt email: Haproxy letsencrypt server: https://acme-v02.api.letsencrypt.org/directory Haproxy log level: ERROR =====> texxolut http-auth information Http auth enabled: false Http auth allowed ips: Http auth users: =====> texxolut letsencrypt information Letsencrypt active: true Letsencrypt autorenew: false Letsencrypt computed dns provider: Letsencrypt global dns provider: Letsencrypt dns provider: Letsencrypt computed email: admin@texxolut.de Letsencrypt global email: admin@texxolut.de Letsencrypt email: Letsencrypt expiration: 1736153414 Letsencrypt computed graceperiod: 2592000 Letsencrypt global graceperiod: Letsencrypt graceperiod: Letsencrypt computed lego docker args: Letsencrypt global lego docker args: Letsencrypt lego docker args: Letsencrypt computed server: https://acme-v02.api.letsencrypt.org/directory Letsencrypt global server: Letsencrypt server: =====> texxolut logs information Logs computed max size: 10m Logs global max size: 10m Logs global vector sink: Logs max size: Logs vector global image: timberio/vector:0.39.0-debian Logs vector sink: =====> texxolut network information Network attach post create: Network attach post deploy: Network bind all interfaces: false Network computed attach post create: Network computed attach post deploy: Network computed bind all interfaces: false Network computed initial network: Network computed tld: Network global attach post create: Network global attach post deploy: Network global bind all interfaces: false Network global initial network: Network global tld: Network initial network: Network static web listener: Network tld: Network web listeners: 172.17.0.6:5000 =====> texxolut nginx information Nginx access log format: Nginx computed access log format: Nginx global access log format: Nginx access log path: Nginx computed access log path: /var/log/nginx/texxolut-access.log Nginx global access log path: /var/log/nginx/texxolut-access.log Nginx bind address ipv4: Nginx computed bind address ipv4: Nginx global bind address ipv4: Nginx bind address ipv6: Nginx computed bind address ipv6: :: Nginx global bind address ipv6: :: Nginx client max body size: Nginx computed client max body size: 1m Nginx global client max body size: 1m Nginx disable custom config: Nginx computed disable custom config: false Nginx global disable custom config: false Nginx error log path: Nginx computed error log path: /var/log/nginx/texxolut-error.log Nginx global error log path: /var/log/nginx/texxolut-error.log Nginx hsts include subdomains: Nginx computed hsts include subdomains: true Nginx global hsts include subdomains: true Nginx hsts max age: Nginx computed hsts max age: 15724800 Nginx global hsts max age: 15724800 Nginx hsts preload: Nginx computed hsts preload: false Nginx global hsts preload: false Nginx hsts: Nginx computed hsts: true Nginx global hsts: true Nginx last visited at: Nginx nginx conf sigil path: Nginx computed nginx conf sigil path: nginx.conf.sigil Nginx global nginx conf sigil path: nginx.conf.sigil Nginx proxy buffer size: Nginx computed proxy buffer size: 4k Nginx global proxy buffer size: 4k Nginx proxy buffering: Nginx computed proxy buffering: on Nginx global proxy buffering: on Nginx proxy buffers: Nginx computed proxy buffers: 8 4k Nginx global proxy buffers: 8 4k Nginx proxy busy buffers size: Nginx computed proxy busy buffers size: 8k Nginx global proxy busy buffers size: 8k Nginx proxy read timeout: Nginx computed proxy read timeout: 60s Nginx global proxy read timeout: 60s Nginx underscore in headers: Nginx computed underscore in headers: off Nginx global underscore in headers: off Nginx x forwarded for value: Nginx computed x forwarded for value: $remote_addr Nginx global x forwarded for value: $remote_addr Nginx x forwarded port value: Nginx computed x forwarded port value: $server_port Nginx global x forwarded port value: $server_port Nginx x forwarded proto value: Nginx computed x forwarded proto value: $scheme Nginx global x forwarded proto value: $scheme Nginx x forwarded ssl: Nginx computed x forwarded ssl: Nginx global x forwarded ssl: =====> texxolut openresty information Openresty access log format: Openresty access log path: /var/log/nginx/texxolut-access.log Openresty allowed letsencrypt domains func base64: cmV0dXJuIHRydWUK Openresty bind address ipv4: Openresty bind address ipv6: :: Openresty client max body size: Openresty error log path: /var/log/nginx/texxolut-error.log Openresty global hsts: true Openresty computed hsts: true Openresty hsts: Openresty hsts include subdomains: true Openresty hsts max age: 15724800 Openresty hsts preload: false Openresty image: dokku/openresty-docker-proxy:0.8.0 Openresty letsencrypt email: Openresty letsencrypt server: https://acme-v02.api.letsencrypt.org/directory Openresty proxy buffer size: 4k Openresty proxy buffering: on Openresty proxy buffers: 8 4k Openresty proxy busy buffers size: 8k Openresty proxy read timeout: 60s Openresty underscore in headers: off Openresty x forwarded for value: $remote_addr Openresty x forwarded port value: $server_port Openresty x forwarded proto value: $scheme Openresty x forwarded ssl: =====> texxolut ports information Ports map: Ports map detected: http:80:5000 https:443:5000 =====> texxolut proxy information Proxy computed type: nginx Proxy enabled: true Proxy global type: nginx Proxy type: =====> texxolut ps information Deployed: true Processes: 1 Ps can scale: true Ps computed procfile path: Procfile Ps global procfile path: Procfile Ps procfile path: Ps restart policy: on-failure:10 Restore: true Running: true Status web 1: running (CID: 6ac8baace6a) =====> texxolut registry information Registry computed image repo: dokku/texxolut Registry computed push on release: false Registry computed server: Registry global image repo template: Registry global push on release: Registry global server: Registry image repo: Registry push extra tags: Registry push on release: Registry server: Registry tag version: =====> texxolut resource information =====> texxolut scheduler information Scheduler computed selected: docker-local Scheduler global selected: docker-local Scheduler selected: =====> texxolut scheduler-docker-local information Scheduler docker local init process: true Scheduler docker local parallel schedule count: =====> texxolut scheduler-k3s information Scheduler k3s computed deploy timeout: 300s Scheduler k3s computed image pull secrets: Scheduler k3s computed letsencrypt server: prod Scheduler k3s computed namespace: default Scheduler k3s computed rollback on failure: false Scheduler k3s deploy timeout: Scheduler k3s global deploy timeout: 300s Scheduler k3s global image pull secrets: Scheduler k3s global ingress class: nginx Scheduler k3s global kube context: Scheduler k3s global kubeconfig path: /etc/rancher/k3s/k3s.yaml Scheduler k3s global letsencrypt email prod: Scheduler k3s global letsencrypt email stag: Scheduler k3s global letsencrypt server: prod Scheduler k3s global namespace: default Scheduler k3s global network interface: eth0 Scheduler k3s global rollback on failure: false Scheduler k3s image pull secrets: Scheduler k3s letsencrypt server: Scheduler k3s namespace: Scheduler k3s rollback on failure: =====> texxolut storage information Storage build mounts: Storage deploy mounts: Storage run mounts: =====> texxolut traefik information Traefik api enabled: false Traefik api vhost: traefik.dokku.me Traefik basic auth password: Traefik basic auth username: Traefik dashboard enabled: false Traefik image: traefik:2.11.2 Traefik letsencrypt email: Traefik letsencrypt server: https://acme-v02.api.letsencrypt.org/directory Traefik log level: ERROR Traefik http entry point: http Traefik https entry point: https ```
josegonzalez commented 1 day ago

[!NOTE] I moved this to the letsencrypt repo

Mind running the following and showing the output of each command?

cat /var/spool/cron/crontabs/dokku
cat /var/log/dokku/letsencrypt.log
BananaAcid commented 1 day ago 
$ cat /var/spool/cron/crontabs/dokku
cat: /var/spool/cron/crontabs/dokku: Permission denied
$ sudo cat /var/spool/cron/crontabs/dokku
[sudo] password for admin: 
cat: /var/spool/cron/crontabs/dokku: No such file or directory

$ cat /var/log/dokku/letsencrypt.log
cat: /var/log/dokku/letsencrypt.log: No such file or directory
josegonzalez commented 1 day ago

Have you deployed any app since the upgrade?

BananaAcid commented 1 day ago

i believe not. But updated existing apps

just updated dokku version to 0.35.5 ... does that help? Same result with the 2 commands above.

BananaAcid commented 1 day ago

... now after updateing, no domain shows the correct app ... but using letsencrypt:enable seems to fix a single app ... hope I can find a command to redo all apps ... did not expect this to get so messy after updating dokku

josegonzalez commented 1 day ago

Oh looks like your report output says auto renew isn't enabled. Maybe just enable that - the command is in the readme.

BananaAcid commented 1 day ago

oh then must have gotten disabled by the update, thanks for finding it.

BananaAcid commented 1 day ago

just as an addition to fixing all domain <-> app mismatches after last update to dokku 0.35.5:

skip_first=true

dokku apps | while IFS= read -r line; do
  if [ "$skip_first" = true ]; then
    skip_first=false
    continue
  fi
  dokku letsencrypt:enable "$line"
done
BananaAcid commented 1 day ago

except 1 .. all point to the correct domain... the one using a docker container (instead of the the others using a node bildpack) uses the domains of an random app from the list ...

any idea to about how to fix this?

josegonzalez commented 1 day ago

Not sure what you mean "all point to the correct domain". Do you mean one domain isnt serving the correct app? Mind checking to see if the app containers are actually running?

BananaAcid commented 1 day ago

Any specific command to check it easily, or should I use the docker container commands usally?

BananaAcid commented 1 day ago

Since domains are configured for each app, each domain should show the content of its app. thats what I mean app pointing to the correct domain. Yeah, technically totally wrong.

The container buildpack one does not show up in sudo docker container ls anymore

How do I get it to work with dokku again?

josegonzalez commented 1 day ago

You can usually check if the app is running as expected via dokku ps:report $APP.

If it's not running, you can dokku ps:rebuild $APP to get it going again.

BananaAcid commented 1 day ago

thanks. Rebooting the server worked as well.