Internal error on SSL certificates when force SSL is active

NginxProxyManager / nginx-proxy-manager

Docker container for managing Nginx proxy hosts with a simple, powerful interface

https://nginxproxymanager.com

MIT License

21.75k stars 2.51k forks source link

Internal error on SSL certificates when force SSL is active #1625

Open Mystery-X opened 2 years ago

Mystery-X commented 2 years ago

[12/2/2021] [3:03:23 PM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
Failed to renew certificate npm-2 with error: Some challenges have failed.
Failed to renew certificate npm-3 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
2 renew failure(s), 0 parse failure(s)

    at ChildProcess.exithandler (node:child_process:397:12)
    at ChildProcess.emit (node:events:390:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)
Connection Error: Error: read ECONNRESET
Connection Error: Error: read ECONNRESET
[12/2/2021] [3:54:36 PM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #3: <**masked**>
[12/2/2021] [3:54:36 PM] [SSL      ] › ℹ  info      Command: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-3" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation 
[12/2/2021] [3:54:39 PM] [Express  ] › ⚠  warning   Command failed: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-3" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation 
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Failed to renew certificate npm-3 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Duplicate relation "access_list" in a relation expression. You should use "a.[b, c]" instead of "[a.b, a.c]". This will cause an error in objection 2.0

When disabling the Force SSL option the renewal went flawless.

[12/2/2021] [3:56:34 PM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #3: <**masked**>
[12/2/2021] [3:56:34 PM] [SSL      ] › ℹ  info      Command: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-3" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation 
[12/2/2021] [3:56:40 PM] [SSL      ] › ℹ  info      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/npm-3.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for <**masked**>

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded: 
  /etc/letsencrypt/live/npm-3/fullchain.pem (success)

So to me it looks like NPM is also trying to forward the http request for cert renewal to SSL and thus it fails to complete the request.

chaptergy commented 2 years ago

Please provide us with the full letsencrypt logs. See https://github.com/jc21/nginx-proxy-manager/issues/1271#user-content-certificate-error

Mystery-X commented 2 years ago

It's not the full, but it contains the proof that it failed to access the file needed todo the verification.

2021-12-02 15:54:39,525:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/<**masked**> HTTP/1.1" 200 1353
2021-12-02 15:54:39,526:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Thu, 02 Dec 2021 15:54:39 GMT
Content-Type: application/json
Content-Length: 1353
Connection: keep-alive
Boulder-Requester: 122098528
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0101KB-iImdk_v4_E8qeaJBpzYY_-RvkALfB9wFV7ilE8Gc
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "<**masked**>"
  },
  "status": "invalid",
  "expires": "2021-12-09T15:54:37Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "Fetching https://<**masked**>/.well-known/acme-challenge/lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY: Error getting validation data",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/<**masked**>/<**masked**>",
      "token": "lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY",
      "validationRecord": [
        {
          "url": "http://<**masked**>/.well-known/acme-challenge/lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY",
          "hostname": "<**masked**>",
          "port": "80",
          "addressesResolved": [
            "<**masked**>"
          ],
          "addressUsed": "<**masked**>"
        },
        {
          "url": "https://<**masked**>/.well-known/acme-challenge/lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY",
          "hostname": "<**masked**>",
          "port": "443",
          "addressesResolved": [
            "<**masked**>"
          ],
          "addressUsed": "<**masked**>"
        }
      ],
      "validated": "2021-12-02T15:54:38Z"
    }
  ]
}
2021-12-02 15:54:39,526:DEBUG:acme.client:Storing nonce: <**masked**>
2021-12-02 15:54:39,526:INFO:certbot._internal.auth_handler:Challenge failed for domain <**masked**>
2021-12-02 15:54:39,526:INFO:certbot._internal.auth_handler:http-01 challenge for <**masked**>
2021-12-02 15:54:39,526:DEBUG:certbot._internal.display.obj:Notifying user: 
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: <**masked**>
  Type:   connection
  Detail: Fetching https://<**masked**>/.well-known/acme-challenge/lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY: Error getting validation data

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

2021-12-02 15:54:39,527:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 178, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2021-12-02 15:54:39,527:DEBUG:certbot._internal.error_handler:Calling registered functions
2021-12-02 15:54:39,527:INFO:certbot._internal.auth_handler:Cleaning up challenges
2021-12-02 15:54:39,527:DEBUG:certbot._internal.plugins.webroot:Removing /data/letsencrypt-acme-challenge/.well-known/acme-challenge/lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY
2021-12-02 15:54:39,527:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2021-12-02 15:54:39,528:ERROR:certbot._internal.renewal:Failed to renew certificate npm-3 with error: Some challenges have failed.
2021-12-02 15:54:39,529:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 475, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1386, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 122, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 335, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 389, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 439, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 178, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2021-12-02 15:54:39,529:DEBUG:certbot._internal.display.obj:Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2021-12-02 15:54:39,529:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2021-12-02 15:54:39,529:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
2021-12-02 15:54:39,529:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2021-12-02 15:54:39,529:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/opt/certbot/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/opt/certbot/lib/python3.7/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1574, in main
    return config.func(config, plugins)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1460, in renew
    renewal.handle_renewal_request(config)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 501, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2021-12-02 15:54:39,530:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)

chaptergy commented 2 years ago

Are you using cloudflare? Does the same error occur if you disable cloudflare?

Mystery-X commented 2 years ago

No there is no cloudflare. But due to your question I think I start to have an idea what's going on... NPM is serving this website for internal use only on port 443, I've only opened port 80 to the outside because I was hopeing this was enought (like certbot) to fetch an SSL cert. But I guess if you enable "Force SSL" it doesn't care if the traffic is going to /.well-known/acme-challenge or not, but instead redirects it always to the SSL port.

Strugglechen1337 commented 2 years ago

Hello, i get this if i try to make a new certificate for my nginx proxy manager proxy host

Error: Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-14" --agree-tos --authenticator webroot --email "" --preferred-challenges "dns,http" --domains "" Saving debug log to /var/log/letsencrypt/letsencrypt.log An unexpected error occurred: There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/ Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

at ChildProcess.exithandler (child_process.js:308:12)
at ChildProcess.emit (events.js:314:20)
at maybeClose (internal/child_process.js:1022:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)

can someone help me? I use nginx proxy manager as docker version on unraid

robertorubioguardia commented 2 years ago

Hi,

Same here, but not just when force SSL is active but all the time. Can't generate nor renew SSL certificates.

Any help will be gratefully thanked.

app_1  | [12/9/2021] [9:12:17 PM] [Nginx    ] › ℹ  info      Reloading Nginx
app_1  | [12/9/2021] [9:12:17 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #92: keylor.srhosting.net
app_1  | [12/9/2021] [9:12:17 PM] [SSL      ] › ℹ  info      Command: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-92" --agree-tos --authenticator webroot --email "soporte@servidoresrapidos.net" --preferred-challenges "dns,http" --domains "keylor.srhosting.net"
app_1  | [12/9/2021] [9:12:17 PM] [Nginx    ] › ℹ  info      Reloading Nginx
app_1  | [12/9/2021] [9:12:18 PM] [Express  ] › ⚠  warning   Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-92" --agree-tos --authenticator webroot --email "soporte@servidoresrapidos.net" --preferred-challenges "dns,http" --domains "keylor.srhosting.net"
app_1  | Another instance of Certbot is already running.
app_1  | Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/tmpyddaiksx/log or re-run Certbot with -v for more details.

the1ts commented 2 years ago

I don''t believe that force SSL is pushing /well-known/acme-challenge to SSL. I'm able to get the configured 404 error when hitting that path on HTTP as is done by the letsencrypt-acme-challenge.conf, any path outside that does redirect to SSL.

It may look like its forcing that URL to SSL if HSTS is turned on and your browser caches that first. This would not be the case for letsencrypt hitting your website for the challenge since its not designed for SSL communications but just plain HTTP so would ignore the HSTS header leaving it on the HTTP connection.

erdoukki commented 2 years ago

Same for me (at first)...! I have checked twice all the Firewall / router redirection to my docker NPM / NextCloud... I have now the check availability working (and green)... But too much try on certificate renewal make it postpone... will try later

Schlumpf9 commented 2 years ago

I have the same problem. When turning on force SSL then Certbot is not able to renew the certificate:

2022-04-26 06:56:14,572:INFO:certbot._internal.auth_handler:Challenge failed for domain XY 2022-04-26 06:56:14,572:INFO:certbot._internal.auth_handler:http-01 challenge for XY 2022-04-26 06:56:14,572:DEBUG:certbot._internal.display.obj:Notifying user: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems: Domain: XY Type: connection Detail: IP: Fetching https://XY/well-known/acme-challenge/lqC8CqFhvzDci89waVFP_4-GgUWqqh273mA6Plv5naI: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

2022-04-26 06:56:14,572:DEBUG:certbot._internal.error_handler:Encountered exception: Traceback (most recent call last): File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 105, in handle_authorizations self._poll_authorizations(authzrs, max_retries, best_effort) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 205, in _poll_authorizations raise errors.AuthorizationError('Some challenges have failed.') certbot.errors.AuthorizationError: Some challenges have failed.

2022-04-26 06:56:14,572:DEBUG:certbot._internal.error_handler:Calling registered functions 2022-04-26 06:56:14,572:INFO:certbot._internal.auth_handler:Cleaning up challenges 2022-04-26 06:56:14,572:DEBUG:certbot._internal.plugins.webroot:Removing /data/letsencrypt-acme-challenge/.well-known/acme-challenge/lqC8CqFhvzDci89waVFP_4-GgUWqqh273mA6Plv5naI 2022-04-26 06:56:14,573:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up 2022-04-26 06:56:14,573:ERROR:certbot._internal.renewal:Failed to renew certificate npm-9 with error: Some challenges have failed. 2022-04-26 06:56:14,573:DEBUG:certbot._internal.renewal:Traceback was: Traceback (most recent call last): File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 485, in handle_renewal_request main.renew_cert(lineage_config, plugins, renewal_candidate) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1441, in renew_cert renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 127, in _get_and_save_cert renewal.renew_cert(config, domains, le_client, lineage) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 345, in renew_cert new_cert, new_chain, newkey, = le_client.obtain_certificate(domains, new_key) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 424, in obtain_certificate orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 476, in _get_order_and_authorizations authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 105, in handle_authorizations self._poll_authorizations(authzrs, max_retries, best_effort) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 205, in _poll_authorizations raise errors.AuthorizationError('Some challenges have failed.') certbot.errors.AuthorizationError: Some challenges have failed.

If i connect to the container and try to curl https://XY/well-known/acme-challenge/lqC8CqFhvzDci89waVFP_4-GgUWqqh273mA6Plv5naI I receive a 404 error so there is no firewall issue there. Requesting http will response with a redirect 301. If i turn off force SSL for the specific domain and try to renew the certificate everything works. So i can definitely agree that forcing SSL prevents certbot from cert renewal... Really annoying -.-

AtryFox commented 2 years ago

I have the same issue here, exactly as described above. As soon as I disable "Force SSL", renewing my certificates works without issues. The renew mechanism should disable "Force SSL" temporarily or add the /well-known/acme-challange/... path as a default rule where SSL is not forced.

the1ts commented 2 years ago

I did notice one difference in config over time. The include of force-ssl.conf is in the server section for newly created hosts, but in the location / section for older hosts. I can break currently working proxy hosts by moving the force-ssl.conf include into the server section, outside the location / section. This change was in #1017, which fixes the custom locations ignoring the force-ssl.conf but appears to override the specific letsencrypt exception to force-ssl. Therefore, I think the test for redirect needs to test both $scheme = "http" and not contains /.well-known/acme-challenge/ As you can't do multiple conditions in one if or nest them, I think this can be done with setting a variable on $scheme = http to H and concatenating a D to the same variable if outside /.well-known/acme-challenge/ so only do the return 301 if the variable = HD.

So we would have:

HTTP and letsencrypt ("H") don't redirect
HTTP and not letsencrypt ("HD") redirect
HTTPS and letsencrypt ("") don't redirect (already HTTPS)
HTTPS and not letsencrypt ("D") don't redirect (already HTTPS)

Guessing here, but we don't see this issues at first creation since the default_host is hit until the cert is obtained and the proxy_host config is written and nginx HUP'd.

n0bbi commented 2 years ago

Same here, if "Force SSL" is enabled, i'm not able to renew the letsencrypt-certificate.

Schlumpf9 commented 2 years ago

lazerlabs commented 2 years ago

lovetox commented 2 years ago

Disabling Force SSL fixed this problem also for me

andriuch commented 2 years ago

Hi Same here, I'm trying to create a new Letsencrypt certificate, with and without Force SSL checked, respond with Internal Server Error, in Nginx Proxy Manager Log is wrote:

[8/25/2022] [1:34:58 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #4: ********.duckdns.org
[8/25/2022] [1:34:58 PM] [SSL      ] › ℹ  info      Command: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-4" --agree-tos --authenticator webroot --email "****@***.com" --preferred-challenges "dns,http" --domains "********.duckdns.org" 
[8/25/2022] [1:35:22 PM] [Nginx    ] › ℹ  info      Reloading Nginx
[8/25/2022] [1:35:22 PM] [Express  ] › ⚠  warning   Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-4" --agree-tos --authenticator webroot --email "****@***.com" --preferred-challenges "dns,http" --domains "********.duckdns.org" 
Saving debug log to /data/logs/letsencrypt/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /data/logs/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

I can't find the logfile /data/logs/letsencrypt/letsencrypt.log

Schlumpf9 commented 1 year ago

Annoying hat this central functionality is still broken :/

EDIflyer commented 1 year ago

Any thoughts on this @jc21 or others? All my subdomain certs are now up for renewal including the one to access npm itself and all are failing...

10/01/2022 7:26:31 PM
[10/2/2022] [2:26:31 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
10/01/2022 7:31:10 PM
[10/2/2022] [2:31:10 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
10/01/2022 7:31:10 PM
Failed to renew certificate npm-1 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
Failed to renew certificate npm-10 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
Failed to renew certificate npm-11 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
Failed to renew certificate npm-2 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
Failed to renew certificate npm-3 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
Failed to renew certificate npm-4 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
Failed to renew certificate npm-5 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
Failed to renew certificate npm-6 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
Failed to renew certificate npm-7 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
Failed to renew certificate npm-8 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
Failed to renew certificate npm-9 with error: Some challenges have failed.
10/01/2022 7:31:10 PM
All renewals failed. The following certificates could not be renewed:
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-10/fullchain.pem (failure)
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-11/fullchain.pem (failure)
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-4/fullchain.pem (failure)
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-5/fullchain.pem (failure)
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-6/fullchain.pem (failure)
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-7/fullchain.pem (failure)
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-8/fullchain.pem (failure)
10/01/2022 7:31:10 PM
  /etc/letsencrypt/live/npm-9/fullchain.pem (failure)
10/01/2022 7:31:10 PM
11 renew failure(s), 0 parse failure(s)
10/01/2022 7:31:10 PM
10/01/2022 7:31:10 PM
    at ChildProcess.exithandler (node:child_process:399:12)
10/01/2022 7:31:10 PM
    at ChildProcess.emit (node:events:526:28)
10/01/2022 7:31:10 PM
    at maybeClose (node:internal/child_process:1092:16)
10/01/2022 7:31:10 PM
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

EDIT: eventually managed to get back into the npm website (blocked by Chrome due to invalid cert, but Firefox let me bypass the warning) and switching off Force SSL let me renew OK, but with 12 sites it's quite a pain to toggle off, renew, then toggle back on!

JulsSkogs commented 1 year ago

I am also experiencing this issue, but even disabling Force SSL changes nothing. I'll try to get a log tomorrow.

EDIflyer commented 1 year ago

So interestingly using :latest I'm still having issues renewing certs but have tried deleting some that wouldn't renew and re-requesting them - they now seem to be renewing OK. Will take a while to re-do them all though!

pierluigizagaria commented 1 year ago

Still having this issue, cannot renew my certificates

EDIflyer commented 1 year ago

I'm now having this issue on another site too. If I delete and recreate they seem to work but renewal has been failing.

Failed to renew certificate npm-2 with error: Some challenges have failed.
Failed to renew certificate npm-3 with error: Some challenges have failed.
Failed to renew certificate npm-7 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-7/fullchain.pem (failure)
4 renew failure(s), 0 parse failure(s)
    at ChildProcess.exithandler (node:child_process:402:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1100:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
[1/20/2023] [1:27:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[1/20/2023] [1:32:20 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
Failed to renew certificate npm-1 with error: Some challenges have failed.
Failed to renew certificate npm-2 with error: Some challenges have failed.
Failed to renew certificate npm-3 with error: Some challenges have failed.
Failed to renew certificate npm-7 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-7/fullchain.pem (failure)
4 renew failure(s), 0 parse failure(s)
    at ChildProcess.exithandler (node:child_process:402:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1100:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
[1/20/2023] [2:27:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[1/20/2023] [2:35:58 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
Failed to renew certificate npm-1 with error: Some challenges have failed.
Failed to renew certificate npm-2 with error: Some challenges have failed.
Failed to renew certificate npm-3 with error: Some challenges have failed.
Failed to renew certificate npm-7 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-7/fullchain.pem (failure)
4 renew failure(s), 0 parse failure(s)
    at ChildProcess.exithandler (node:child_process:402:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1100:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
[1/20/2023] [3:27:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[1/20/2023] [3:31:33 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
Failed to renew certificate npm-1 with error: Some challenges have failed.
Failed to renew certificate npm-2 with error: Some challenges have failed.
Failed to renew certificate npm-3 with error: Some challenges have failed.
Failed to renew certificate npm-7 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-7/fullchain.pem (failure)
4 renew failure(s), 0 parse failure(s)
    at ChildProcess.exithandler (node:child_process:402:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1100:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
[1/20/2023] [4:22:52 AM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #1: npm.***
[1/20/2023] [4:22:52 AM] [SSL      ] › ℹ  info      Command: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-1" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation 
[1/20/2023] [4:23:18 AM] [Express  ] › ⚠  warning   Command failed: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-1" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation 
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Failed to renew certificate npm-1 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
[1/20/2023] [4:24:04 AM] [SSL      ] › ℹ  info      Revoking Let'sEncrypt certificates for Cert #2: logs.***
[1/20/2023] [4:24:04 AM] [SSL      ] › ℹ  info      Command: certbot revoke --config "/etc/letsencrypt.ini" --cert-path "/etc/letsencrypt/live/npm-2/fullchain.pem" --delete-after-revoke ; rm -f '/etc/letsencrypt/credentials/credentials-2' || true
[1/20/2023] [4:24:06 AM] [SSL      ] › ℹ  info      Deleted all files relating to certificate npm-2.
Congratulations! You have successfully revoked the certificate that was located at /etc/letsencrypt/live/npm-2/fullchain.pem.
Duplicate relation "access_list" in a relation expression. You should use "a.[b, c]" instead of "[a.b, a.c]". This will cause an error in objection 2.0
[1/20/2023] [4:24:22 AM] [Nginx    ] › ℹ  info      Reloading Nginx
[1/20/2023] [4:24:27 AM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #8: logs.***
[1/20/2023] [4:24:27 AM] [SSL      ] › ℹ  info      Command: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-8" --agree-tos --authenticator webroot --email "webmaster@***" --preferred-challenges "dns,http" --domains "***" 
[1/20/2023] [4:24:44 AM] [SSL      ] › ✔  success   Requesting a certificate for npm.***
Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/npm-8/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/npm-8/privkey.pem
This certificate expires on 2023-04-20.
These files will be updated when the certificate renews.
NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[1/20/2023] [4:24:44 AM] [Nginx    ] › ℹ  info      Reloading Nginx
[1/20/2023] [4:24:45 AM] [Nginx    ] › ℹ  info      Reloading Nginx

@jc21 would really appreciate any help here - I keep on having to delete and recreate certs from scratch which with lots of subdomains can take quite a while! Weirdly the other site where I recreated them still seems to be renewing OK?

EDIflyer commented 1 year ago

There also seems to be an issue when deleting certificates too (from within the interface!) as end up with these sorts of errors:

01/20/2023 12:34:54 PM
[1/20/2023] [4:34:54 AM] [Express  ] › ⚠  warning   Command failed: /usr/sbin/nginx -t -g "error_log off;"
01/20/2023 12:34:54 PM
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-3/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/npm-3/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
01/20/2023 12:34:54 PM
nginx: configuration file /etc/nginx/nginx.conf test failed
01/20/2023 12:34:54 PM
01/20/2023 12:34:58 PM
[1/20/2023] [4:34:58 AM] [Express  ] › ⚠  warning   Command failed: /usr/sbin/nginx -t -g "error_log off;"
01/20/2023 12:34:58 PM
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-3/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/npm-3/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
01/20/2023 12:34:58 PM
nginx: configuration file /etc/nginx/nginx.conf test failed
01/20/2023 12:34:58 PM
01/20/2023 12:35:35 PM
[1/20/2023] [4:35:35 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
01/20/2023 12:35:35 PM
Failed to renew certificate npm-1 with error: Some challenges have failed.
01/20/2023 12:35:35 PM
Renewal configuration file /etc/letsencrypt/renewal/npm-3.conf is broken.
01/20/2023 12:35:35 PM
The error was: renewal config file {} is missing a required file reference
01/20/2023 12:35:35 PM
Skipping.
01/20/2023 12:35:35 PM
Renewal configuration file /etc/letsencrypt/renewal/npm-5.conf is broken.
01/20/2023 12:35:35 PM
The error was: renewal config file {} is missing a required file reference
01/20/2023 12:35:35 PM
Skipping.
01/20/2023 12:35:35 PM
Renewal configuration file /etc/letsencrypt/renewal/npm-7.conf is broken.
01/20/2023 12:35:35 PM
The error was: renewal config file {} is missing a required file reference
01/20/2023 12:35:35 PM
Skipping.
01/20/2023 12:35:35 PM
All renewals failed. The following certificates could not be renewed:
01/20/2023 12:35:35 PM
  /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
01/20/2023 12:35:35 PM
1 renew failure(s), 3 parse failure(s)
01/20/2023 12:35:35 PM
01/20/2023 12:35:35 PM
    at ChildProcess.exithandler (node:child_process:402:12)
01/20/2023 12:35:35 PM
    at ChildProcess.emit (node:events:513:28)
01/20/2023 12:35:35 PM
    at maybeClose (node:internal/child_process:1100:16)
01/20/2023 12:35:35 PM
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
01/20/2023 12:35:49 PM
[1/20/2023] [4:35:49 AM] [Express  ] › ⚠  warning   Command failed: /usr/sbin/nginx -t -g "error_log off;"
01/20/2023 12:35:49 PM
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-3/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/npm-3/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
01/20/2023 12:35:49 PM
nginx: configuration file /etc/nginx/nginx.conf test failed

I've found copying existing good directories across to the missing ones then allows re-creation but it seems like the nginx config isn't updated when a cert is deleted? Workaround seems to be to create a new certificate and then delete the old one.

EDIflyer commented 1 year ago

Any update on this @jc21 ?

I'm running two servers and one of them seems to be OK...

12/02/2023 10:27:15
[2/12/2023] [10:27:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
12/02/2023 10:27:18
[2/12/2023] [10:27:18 AM] [Nginx    ] › ℹ  info      Reloading Nginx
12/02/2023 10:27:18
[2/12/2023] [10:27:18 AM] [SSL      ] › ℹ  info      Renew Complete
12/02/2023 11:27:14
[2/12/2023] [11:27:14 AM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
12/02/2023 11:27:14
[2/12/2023] [11:27:14 AM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
12/02/2023 11:27:14
[2/12/2023] [11:27:14 AM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
12/02/2023 11:27:14
[2/12/2023] [11:27:14 AM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
12/02/2023 11:27:14
[2/12/2023] [11:27:14 AM] [Nginx    ] › ℹ  info      Reloading Nginx
12/02/2023 11:27:15
[2/12/2023] [11:27:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
12/02/2023 11:27:16
[2/12/2023] [11:27:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx
12/02/2023 11:27:17
[2/12/2023] [11:27:17 AM] [SSL      ] › ℹ  info      Renew Complete

The other still has errors...

12/02/2023 12:05:46
Failed to renew certificate npm-17 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-18 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-26 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-29 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-30 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-31 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-32 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-33 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-34 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-35 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-36 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-37 with error: Some challenges have failed.
12/02/2023 12:05:46
Failed to renew certificate npm-38 with error: Some challenges have failed.
12/02/2023 12:05:46
All renewals failed. The following certificates could not be renewed:
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-17/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-18/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-26/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-29/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-30/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-31/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-32/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-33/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-34/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-35/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-36/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-37/fullchain.pem (failure)
12/02/2023 12:05:46
  /etc/letsencrypt/live/npm-38/fullchain.pem (failure)
12/02/2023 12:05:46
13 renew failure(s), 0 parse failure(s)
12/02/2023 12:05:46
12/02/2023 12:05:46
    at ChildProcess.exithandler (node:child_process:402:12)
12/02/2023 12:05:46
    at ChildProcess.emit (node:events:513:28)
12/02/2023 12:05:46
    at maybeClose (node:internal/child_process:1100:16)
12/02/2023 12:05:46
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)

Yet it was the other way round previously. It's like they get stuck renewing at some point and then that's it!

github-actions[bot] commented 6 months ago

Issue is now considered stale. If you want to keep it open, please comment :+1:

rushhee commented 6 months ago

Did this ever get addressed?

On Thu, 29 Feb 2024, 12:48 pm github-actions[bot], @.***> wrote:

Issue is now considered stale. If you want to keep it open, please comment 👍

— Reply to this email directly, view it on GitHub https://github.com/NginxProxyManager/nginx-proxy-manager/issues/1625#issuecomment-1970246539, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3I7NSSTCLIEX3YKTPE5TU3YV2EFLAVCNFSM5JHQ3PY2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJXGAZDINRVGM4Q . You are receiving this because you are subscribed to this thread.Message ID: @.*** com>

gabrio79 commented 5 months ago

any news?