search

NginxProxyManager / nginx-proxy-manager

Docker container for managing Nginx proxy hosts with a simple, powerful interface
https://nginxproxymanager.com
MIT License
23.01k stars 2.67k forks source link

All certificate renewals suddenly fail. Has worked for months if not years. #2267

Open Waldorf3 opened 2 years ago

Waldorf3 commented 2 years ago

After using this container for months, if not years, with minimal interaction necessary, suddenly certificates are no longer automatically updated. Nothing changed in my environment so I'm a bit at a loss what happened.

From the log:

nginx-proxy-manager-app-1  | [9/15/2022] [4:50:19 PM] [Global   ] › ℹ  info      Generating MySQL knex configuration from environment variables
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:19 PM] [Global   ] › ⬤  debug     Wrote db configuration to config file: ./config/production.json
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:20 PM] [Migrate  ] › ℹ  info      Current database version: 20211108145214
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:20 PM] [Setup    ] › ℹ  info      Creating a new JWT key pair...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Setup    ] › ℹ  info      Wrote JWT key pair to config file: /app/config/production.json
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Setup    ] › ℹ  info      Logrotate Timer initialized
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Setup    ] › ℹ  info      Logrotate completed.
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Global   ] › ℹ  info      Backend PID 246 listening on port 3000 ...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:49 PM] [Express  ] › ⚠  warning   invalid signature
nginx-proxy-manager-app-1  | [9/15/2022] [4:52:43 PM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
nginx-proxy-manager-app-1  | Failed to renew certificate npm-1 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-10 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-11 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-12 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-13 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-2 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-25 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-3 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-31 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-32 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-34 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-35 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-36 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-38 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-39 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-4 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-40 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-41 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-42 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-5 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-6 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-7 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-8 with error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/
nginx-proxy-manager-app-1  | Failed to renew certificate npm-9 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | All renewals failed. The following certificates could not be renewed:
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-10/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-11/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-12/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-13/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-25/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-31/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-32/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-34/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-35/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-36/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-38/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-39/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-4/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-40/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-41/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-42/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-5/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-6/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-7/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-8/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-9/fullchain.pem (failure)
nginx-proxy-manager-app-1  | 24 renew failure(s), 0 parse failure(s)
nginx-proxy-manager-app-1  |
nginx-proxy-manager-app-1  |     at ChildProcess.exithandler (node:child_process:399:12)
nginx-proxy-manager-app-1  |     at ChildProcess.emit (node:events:526:28)
nginx-proxy-manager-app-1  |     at maybeClose (node:internal/child_process:1092:16)
nginx-proxy-manager-app-1  |     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
nginx-proxy-manager-app-1  | `QueryBuilder#allowEager` method is deprecated. You should use `allowGraph` instead. `allowEager` method will be removed in 3.0
nginx-proxy-manager-app-1  | `QueryBuilder#eager` method is deprecated. You should use the `withGraphFetched` method instead. `eager` method will be removed in 3.0
nginx-proxy-manager-app-1  | QueryBuilder#omit is deprecated. This method will be removed in version 3.0
nginx-proxy-manager-app-1  | Model#$omit is deprected and will be removed in 3.0.
nginx-proxy-manager-app-1  | Duplicate relation "access_list" in a relation expression. You should use "a.[b, c]" instead of "[a.b, a.c]". This will cause an error in objection 2.0
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:08 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:30 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:53 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:58 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #46: myapp.mydomain.com
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:58 PM] [SSL      ] › ℹ  info      Command: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-46" --agree-tos --authenticator webroot --email "me@mydomain.com" --preferred-challenges "dns,http" --domains "myapp.mydomain.com"
nginx-proxy-manager-app-1  | [9/15/2022] [4:56:05 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:56:05 PM] [Express  ] › ⚠  warning   Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-46" --agree-tos --authenticator webroot --email "me@mydomain.com" --preferred-challenges "dns,http" --domains "myapp.mydomain.com"
nginx-proxy-manager-app-1  | Saving debug log to /var/log/letsencrypt/letsencrypt.log
nginx-proxy-manager-app-1  | Some challenges have failed.
nginx-proxy-manager-app-1  | Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
the1ts commented 2 years ago

@Waldorf3 There is a problem with renewing if force SSL is turned on, if that is true for you, turn off force SSL, renew then turn back on. Its awaiting a fix to be merged #2038

huntitus commented 2 years ago

We have the same symptoms. The certificates not renewed by automatically. When I try to renew manually on the GUI, it provide an "internal error" message (without any error code, or details). Sadly this solution is not reliable :/

Edit:

Looks like this problem is very old :/ and still exist.

Waldorf3 commented 2 years ago

We have the same symptoms. The certificates not renewed by automatically. When I try to renew manually on the GUI, it provide an "internal error" message (without any error code, or details). Sadly this solution is not reliable :/

Edit:

Looks like this problem is very old :/ and still exist.

I already tried the "uncheck force ssl and reissue cert", it just throws an "internal error".

I'm running NPM in docker, not sure how to start fiddling with a script to fix this. Would be better if the author would acknowledge the bug and offer a proper solution.

Waldorf3 commented 2 years ago

I'm just realizing I might be barking up the wrong tree. Is jc21 the actual developer/maintainer of this code, or just the docker-packager? If not, who is responsible for the code, who can fix this bug?

I actually find the NPM SSL subsystem to be quite fragile. If for example you try to enable SSL for a site that does not have a root directory, such as for example ubooquity (requires http://ubooquity/ubooquity) it will also fail with a nondescript "internal error", and only way to fix is manually cleaning up the database. That's a viable solution for a single failed certificate, not for a system with 30 or more proxy hosts with failed certs.

the1ts commented 2 years ago

JC21 is the developer and the only fragility at the moment that I see is the renew issue outstanding a merge. I simply don't see how NPM fronting for ubooquity results in an internal error that requires DB cleanup. The application isn't even hit for letsencrypt SSL certs to be created via the HTTP auth method. Also picking a pretty broken by modern standards application like ubooquity to measure success is odd. It has, as you say, no root directory so requires special measures and also a different port for admin so requires special measures again, no tool calling itself easy to use can be expected to handle both those broken by modern standards decisions. Often internal errors are after people remove active SSL certs breaking Nginx config i.e. ignoring the warning. What specific errors are you still getting?

EDIflyer commented 2 years ago

Just to confirm that PR #2038 by @the1ts seems to be doing the trick for me.

github-actions[bot] commented 9 months ago

Issue is now considered stale. If you want to keep it open, please comment :+1:

Saibamen commented 9 months ago

👍

jdhorner commented 5 days ago

👍