search

NginxProxyManager / nginx-proxy-manager

Docker container for managing Nginx proxy hosts with a simple, powerful interface
https://nginxproxymanager.com
MIT License
21.91k stars 2.53k forks source link

Error when generating the SSL certificate (no Challenge!) #3250

Open javierspn opened 11 months ago

javierspn commented 11 months ago

Checklist

Describe the bug

When I try to generate an SSL certificate suing the npm web ui, and I clicke "Test Server reachability", the following message is displayed: Communication with the API failed, is NPM running correctly? I must add that I cannot use DNS challenge as one.com is my DNS provider and it has no plugin on the drop down list.

Nginx Proxy Manager Version

v2.10.4

To Reproduce Steps to reproduce the behavior:

  1. Go to 'SSL Certificates'
  2. Click on 'Add SSL certificate'
  3. Enter domain name, in my case: xxx.yyyyy.com
  4. Message is displayed.

Expected behavior

Well to contact the DNS domain and generate the SSL certificate obviously.

Screenshots

Operating System

Ubuntu 22.04.3

Additional context

  1. The machine is an Oracle Cloud Instance.
  2. DNS record (A type) is properly registered and replies with the public IP of the Oracle Instance (where it is pointing).
  3. Replication tested with 24x7.com tools.
  4. nslookup executed on the docker host and a remote client, DNS is properly resolved.
  5. Firewall on the machine opened for ports 80/81/443 and 22 for management.
  6. Rules on the OCI cloud (for this machine) open for the ports above too (and tested!).

Log from the container:

bash: line 1: 299 Trace/breakpoint trap (core dumped) node --abort_on_uncaught_exception --max_old_space_size=250 index.js ❯ Starting backend ... [10/9/2023] [1:59:56 PM] [Global ] › ℹ info Using Sqlite: /data/database.sqlite [10/9/2023] [1:59:57 PM] [Migrate ] › ℹ info Current database version: none [10/9/2023] [1:59:57 PM] [Setup ] › ℹ info Logrotate Timer initialized [10/9/2023] [1:59:57 PM] [Setup ] › ℹ info Logrotate completed. [10/9/2023] [1:59:57 PM] [IP Ranges] › ℹ info Fetching IP Ranges from online services... [10/9/2023] [1:59:57 PM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json [10/9/2023] [1:59:57 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4 [10/9/2023] [1:59:57 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6 [10/9/2023] [1:59:57 PM] [SSL ] › ℹ info Let's Encrypt Renewal Timer initialized [10/9/2023] [1:59:57 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry... [10/9/2023] [1:59:57 PM] [IP Ranges] › ℹ info IP Ranges Renewal Timer initialized [10/9/2023] [1:59:57 PM] [Global ] › ℹ info Backend PID 329 listening on port 3000 ... [10/9/2023] [1:59:58 PM] [Nginx ] › ℹ info Reloading Nginx [10/9/2023] [1:59:58 PM] [SSL ] › ℹ info Renew Complete [10/9/2023] [2:06:27 PM] [SSL ] › ℹ info Testing http challenge for xxx.yyyyy.com <-- (edited for privacy purposes) by me Uncaught SyntaxError: Unexpected end of JSON input

giovannicimolin commented 11 months ago

I just updated my install and I'm getting the same error.

giovannicimolin commented 11 months ago

After a bit of digging:

  1. The test relies on https://www.site24x7.com/tools/restapi-tester in this method: https://github.com/NginxProxyManager/nginx-proxy-manager/blob/3197de41de89786a7fb73a61fbf3f1e271e03091/backend/internal/certificate.js#L1147
  2. That address is currently returning an error: https://www.site24x7.com/tools/restapi-tester (server error). The tool itself seems to be working though.
  3. It tries to parse the response as JSON, and that leads to an uncaught exception here: https://github.com/NginxProxyManager/nginx-proxy-manager/blob/3197de41de89786a7fb73a61fbf3f1e271e03091/backend/internal/certificate.js#L1182
giovannicimolin commented 11 months ago

CC @jc21 as repo maintainer.

1xtr commented 11 months ago

Today I have installed npm on new VM and have same issue. i can't add Let's certs and server test reachability don't work

giovannicimolin commented 11 months ago

@1xtr Hi! You don't need do to the reachability test to get the certificate. You can issue certificates without it (just click save on that modal when setting up the certificate).

secdoc commented 10 months ago

I am getting the issue when trying to generate a cert through Let's Encrypt. It was working yesterday after doing the initial

install and as of today. I am currently running:

NPM_BUILD_VERSION | 2.10.4

Installed on Proxmox VM: OS: Ubuntu 20.04.6 LTS x86_64 Host: KVM/QEMU (Standard PC (Q35 + ICH9, 2009) pc-q35-8.0) CPU: AMD Ryzen 9 5950X (4) @ 3.399GHz Memory: 472MiB / 3919MiB npm-error

NPM Console Log: `2023-10-19T13:13:59.053188248Z [10/19/2023] [1:13:59 PM] [SSL ] › ℹ info Requesting Let'sEncrypt certificates for Cert #22: shield.secdoc.tech

2023-10-19T13:13:59.053213318Z [10/19/2023] [1:13:59 PM] [SSL ] › ℹ info Command: certbot certonly --config "/etc/letsencrypt.ini" --work-dir "/tmp/letsencrypt-lib" --logs-dir "/tmp/letsencrypt-log" --cert-name "npm-22" --agree-tos --authenticator webroot --email "secdoc@protonmail.com" --preferred-challenges "dns,http" --domains "shield.secdoc.tech"

2023-10-19T13:13:59.517255881Z [10/19/2023] [1:13:59 PM] [Nginx ] › ⬤ debug Deleting file: /data/nginx/temp/letsencrypt_22.conf

2023-10-19T13:13:59.548989227Z [10/19/2023] [1:13:59 PM] [Nginx ] › ℹ info Reloading Nginx

2023-10-19T13:13:59.575031046Z [10/19/2023] [1:13:59 PM] [Express ] › ⚠ warning Command failed: certbot certonly --config "/etc/letsencrypt.ini" --work-dir "/tmp/letsencrypt-lib" --logs-dir "/tmp/letsencrypt-log" --cert-name "npm-22" --agree-tos --authenticator webroot --email "secdoc@protonmail.com" --preferred-challenges "dns,http" --domains "shield.secdoc.tech"

2023-10-19T13:13:59.575049576Z Saving debug log to /tmp/letsencrypt-log/letsencrypt.log

2023-10-19T13:13:59.575052366Z An unexpected error occurred:

2023-10-19T13:13:59.575054426Z Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/

2023-10-19T13:13:59.575056386Z Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/letsencrypt-log/letsencrypt.log or re-run Certbot with -v for more details.`

Container Console running Cert Renewal Dry Run: `[root@docker-e931ae59842d:/app]# certbot renew --dry-run Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/npm-1.conf

Account registered. Simulating renewal of an existing certificate for cerebro.secdoc.tech

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems: Domain: cerebro.secdoc.tech Type: connection Detail: 99.92.243.109: Fetching https://cerebro.secdoc.tech/.well-known/acme-challenge/4lMd_By6EoJRTsDWOs_1AClnJR_BVGO-7GbyHYglv84: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Failed to renew certificate npm-1 with error: Some challenges have failed.

Processing /etc/letsencrypt/renewal/npm-2.conf

Simulating renewal of an existing certificate for rpi-hosted.secdoc.tech

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems: Domain: rpi-hosted.secdoc.tech Type: connection Detail: 99.92.243.109: Fetching https://rpi-hosted.secdoc.tech/.well-known/acme-challenge/VFHW3u7pidVAeDUmRWXhdWkx0mAE8o1S8VXeHIECii0: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Failed to renew certificate npm-2 with error: Some challenges have failed.

All simulated renewals failed. The following certificates could not be renewed: /etc/letsencrypt/live/npm-1/fullchain.pem (failure) /etc/letsencrypt/live/npm-2/fullchain.pem (failure)

2 renew failure(s), 0 parse failure(s) Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.`

Here is the debug log from let's encrypt: `[root@docker-e931ae59842d:/app]# cat /tmp/letsencrypt-log/letsencrypt.log | more 2023-10-19 13:13:59,227:DEBUG:certbot._internal.main:certbot version: 2.5.0 2023-10-19 13:13:59,227:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot 2023-10-19 13:13:59,227:DEBUG:certbot._internal.main:Arguments: ['--config', '/etc/letsencrypt.ini', '--work-dir', '/tmp/letsencrypt-lib', '--logs-d ir', '/tmp/letsencrypt-log', '--cert-name', 'npm-22', '--agree-tos', '--authenticator', 'webroot', '--email', 'secdoc@protonmail.com', '--preferred- challenges', 'dns,http', '--domains', 'shield.secdoc.tech'] 2023-10-19 13:13:59,227:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPo int#standalone,PluginEntryPoint#webroot) 2023-10-19 13:13:59,235:DEBUG:certbot._internal.log:Root logging level set at 30 2023-10-19 13:13:59,235:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None 2023-10-19 13:13:59,237:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * webroot Description: Saves the necessary validation files to a .well-known/acme-challenge/ directory within the nominated webroot path. A seperate HTTP serv er must be running and serving files from the webroot path. HTTP challenge only (wildcards not supported). Interfaces: Authenticator, Plugin Entry point: webroot = certbot._internal.plugins.webroot:Authenticator Initialized: <certbot._internal.plugins.webroot.Authenticator object at 0x7f48af95df98> Prep: True 2023-10-19 13:13:59,237:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.webroot.Authenticator object at 0x7f48af95df98> and installer None 2023-10-19 13:13:59,237:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator webroot, Installer None 2023-10-19 13:13:59,248:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement =None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.o rg/acme/acct/1367521256', new_authzr_uri=None, terms_of_service=None), ca8d13776e9c6038dd9a25a09955dcee, Meta(creation_dt=datetime.datetime(2023, 10 , 18, 18, 39, 57, tzinfo=), creation_host='e931ae59842d', register_to_eff=None))> 2023-10-19 13:13:59,248:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory. 2023-10-19 13:13:59,249:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443 2023-10-19 13:13:59,390:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 752 2023-10-19 13:13:59,391:DEBUG:acme.client:Received response: HTTP 200 Server: nginx Date: Thu, 19 Oct 2023 13:13:59 GMT Content-Type: application/json Content-Length: 752 Connection: keep-alive Cache-Control: public, max-age=0, no-cache X-Frame-Options: DENY Strict-Transport-Security: max-age=604800

{ "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change", "meta": { "caaIdentities": [ "letsencrypt.org" ], "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf", "website": "https://letsencrypt.org" }, "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct", "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce", "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order", "oilU1WS993A": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417", "renewalInfo": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-01/renewalInfo/", "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert" } 2023-10-19 13:13:59,391:DEBUG:certbot._internal.display.obj:Notifying user: Requesting a certificate for shield.secdoc.tech 2023-10-19 13:13:59,393:DEBUG:acme.client:Requesting fresh nonce 2023-10-19 13:13:59,393:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce. 2023-10-19 13:13:59,434:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0 2023-10-19 13:13:59,434:DEBUG:acme.client:Received response: HTTP 200 Server: nginx Date: Thu, 19 Oct 2023 13:13:59 GMT Connection: keep-alive Cache-Control: public, max-age=0, no-cache Link: https://acme-v02.api.letsencrypt.org/directory;rel="index" Replay-Nonce: _s_5u1NQZ3SDcQcMPGYfi2q0ljYyeXvldnR-hzxHD6IEOjnkess X-Frame-Options: DENY Strict-Transport-Security: max-age=604800

2023-10-19 13:13:59,434:DEBUG:acme.client:Storing nonce: _s_5u1NQZ3SDcQcMPGYfi2q0ljYyeXvldnR-hzxHD6IEOjnkess 2023-10-19 13:13:59,434:DEBUG:acme.client:JWS payload: b'{\n "identifiers": [\n {\n "type": "dns",\n "value": "shield.secdoc.tech"\n }\n ]\n}' 2023-10-19 13:13:59,435:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order: { "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTM2NzUyMTI1NiIsICJub25jZSI6ICJfc181 dTFOUVozU0RjUWNNUEdZZmkycTBsall5ZVh2bGRuUi1oenhIRDZJRU9qbmtlc3MiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1vcmRlciJ9 ", "signature": "d_PczDBa5DvGFDO9LOf4h5kLM4xlnsr3OfhHOs8g1RP0G-41tVh5S41LqsqgOBzV7eL6-g4-pXNYRiuTin71FNaJ34JcpPcZ2N4ZBc-LTpB6lmzaKS9Ft6xAfnTd17EomiiN tS-PhLcO8t3Ul0VEWEEmggDE0tEb_dOy3OAmf0T9Ug5xS7H11JISxnyg3C5BIkCrL4Ges8UB2DwmZk2_51gPn5DoePcJY2qGnGriYFO3ljtpZhbnSt2cGEauqF6hQkhTgTSoUXgJ1M59BjKzeU6v PtotV9AmLkpqccyP1pz79fEWIC0OmOn6asL0K7szlakOfifPeXLKlnFJ1Tn5Uw", "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogInNoaWVsZC5zZWNkb2MudGVjaCIKICAgIH0KICBdCn0" } 2023-10-19 13:13:59,495:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 429 213 2023-10-19 13:13:59,495:DEBUG:acme.client:Received response: HTTP 429 Server: nginx Date: Thu, 19 Oct 2023 13:13:59 GMT Content-Type: application/problem+json Content-Length: 213 Connection: keep-alive Boulder-Requester: 1367521256 Cache-Control: public, max-age=0, no-cache Link: https://acme-v02.api.letsencrypt.org/directory;rel="index" Replay-Nonce: _s_5u1NQR_0CvuIseUinfhoDmI_tKZAs-k33T3kaqbamLgkDvLg

{ "type": "urn:ietf:params:acme:error:rateLimited", "detail": "Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/", "status": 429 } 2023-10-19 13:13:59,496:DEBUG:certbot._internal.log:Exiting abnormally: Traceback (most recent call last): File "/usr/bin/certbot", line 8, in sys.exit(main()) File "/opt/certbot/lib/python3.7/site-packages/certbot/main.py", line 19, in main return internal_main.main(cli_args) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1864, in main return config.func(config, plugins) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1597, in certonly lineage = _get_and_save_cert(le_client, config, domains, certname, lineage) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 141, in _get_and_save_cert lineage = le_client.obtain_and_enroll_certificate(domains, certname) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 516, in obtain_and_enrollcertificate cert, chain, key, = self.obtain_certificate(domains) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 428, in obtain_certificate orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names) File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 478, in _get_order_and_authorizations orderr = self.acme.new_order(csr_pem) File "/opt/certbot/lib/python3.7/site-packages/acme/client.py", line 138, in new_order response = self._post(self.directory['newOrder'], order) File "/opt/certbot/lib/python3.7/site-packages/acme/client.py", line 366, in _post return self.net.post(*args, *kwargs) File "/opt/certbot/lib/python3.7/site-packages/acme/client.py", line 739, in post return self._post_once(args, **kwargs) File "/opt/certbot/lib/python3.7/site-packages/acme/client.py", line 752, in _post_once response = self._check_response(response, content_type=content_type) File "/opt/certbot/lib/python3.7/site-packages/acme/client.py", line 603, in _check_response raise messages.Error.from_json(jobj) acme.messages.Error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/ 2023-10-19 13:13:59,496:ERROR:certbot._internal.log:An unexpected error occurred: 2023-10-19 13:13:59,496:ERROR:certbot._internal.log:Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org /docs/failed-validation-limit/`

So I am not doing any DNS check....

Based on the Le's Encrypt log: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/

According to https://letsencrypt.org/docs/failed-validation-limit/

All issuance requests are subject to a Failed Validation limit of 5 failures per account, per hostname, per hour. You should receive the following error message from your ACME client when you’ve exceeded the Failed Validation limit:

too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/

The ‘authorizations’ that this error refers to are the result of authorization requests, sent by your ACME client, to validate control over a domain name before we can issue or renew a certificate. This error indicates that the multiple requests for validation were sent successfully but all attempts to validate have failed. Common Causes

Subscribers who hit the Failed Validation limit often do so due to a misconfiguration in their environment. HTTP-01 or TLS-APLN-01

For ACME clients requesting authorization via the HTTP-01 or TLS-APLN-01 validation methods, the problem usually stems from a network or firewall configuration which makes it impossible for our validation servers to reach the server that the request was sent from. DNS-01

ACME clients requesting authorization via the DNS-01 validation method usually require that you create a CNAME record in your main DNS zone which allows the ACME client to set the required DNS records during the validation process. Failed DNS-01 validations are usually the result of missed steps or typos during this initial setup process.

When troubleshooting or testing the deployment of your applications we encourage you to configure your ACME client to use our staging environment. Rate limits for our staging environment are significantly higher.

So, if this is the root cause of the failure, what are the potential options to configure, knowing that the "network" issue was not an issue yesterday and there have been no changes other than the configuration of NPM Proxy Hosts? Thoughts...

EDIflyer commented 10 months ago

It might be worth trying again using the PR I created at https://github.com/NginxProxyManager/nginx-proxy-manager/pull/3121 and see if that does the trick - has been working for me for a few months now.

github-actions[bot] commented 3 months ago

Issue is now considered stale. If you want to keep it open, please comment :+1: