Internal error when requesting new certificate

[EricGuic]

Hi, I get an "internal error" when I try to request a ssl certificate after setting up a proxy host. It was working fine in july (I've created multiple host with working certificate the 16th july 2021), but now it seems to be broken. I'm running NPM on a raspberry pi, inside Docker (alongside Watchtower, AdGuard Home and Portainer other containers). Being not very good at docker things and ssh command, I manage my container with Portainer GUI.

I've try to find a similar issue here, but without succes. Here are the first clues I can provide to you (sorry for my english from France).

My router is ok, 80 and 443 redirected to my Pi. I can reach some services from outside my network.

So my images : jc21/nginx-proxy-manager:latest yobasystems/alpine-mariadb:10.4.17-arm32v7 (I try with :latest, but never get it working. I finally found some advice online saying that on the raspberry pi, you have to use this version. It works for me, but if you have advice, I will take it.

NPM container settings :

When I try to get the certificate, here are the logs inside Portainer :

[8/30/2021] [8:31:40 PM] [Nginx] › ℹ  info   Reloading Nginx
[8/30/2021] [8:31:40 PM] [SSL] › ℹ  info   Requesting Let'sEncrypt certificates for Cert #22: rss.mydomain.fr
[8/30/2021] [8:31:40 PM] [SSL] › ℹ  info   Command: certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-22" --agree-tos --authenticator webroot --email "my@mail.com" --preferred-challenges "dns,http" --domains "rss.mydomain.fr"
[8/30/2021] [8:31:52 PM] [Nginx    ] › ℹ  info   Reloading Nginx
[8/30/2021] [8:31:52 PM] [Express  ] › ⚠  warning   Command failed: certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-22" --agree-tos --authenticator webroot --email "my@mail.com" --preferred-challenges "dns,http" --domains "rss.mydomain.fr"

Saving debug log to /var/log/letsencrypt/letsencrypt.log

An unexpected error occurred:
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xb59d5eb0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Inside the console, if I run certbot renew, I get :

[root@docker-f244e925705b:/app]# certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Processing /etc/letsencrypt/renewal/npm-1.conf
Certificate not yet due for renewal
Processing /etc/letsencrypt/renewal/npm-2.conf
Certificate not yet due for renewal
Processing /etc/letsencrypt/renewal/npm-3.conf
Certificate not yet due for renewal

The following certificates are not due for renewal yet:
  /etc/letsencrypt/live/npm-1/fullchain.pem expires on 2021-10-14 (skipped)
  /etc/letsencrypt/live/npm-2/fullchain.pem expires on 2021-10-14 (skipped)
  /etc/letsencrypt/live/npm-3/fullchain.pem expires on 2021-10-14 (skipped)
No renewals were attempted.

Those certificate correspond (I guess) to the working certificate I've already set up the 16th july 2021 and which are valid until october.

After that, I'm a bit lost. I don't know how to reach the Lets'Encrypt log, or other specific log which could be useful.

Thanks for your help.

(edit : clarify syntax, anonymised personnal email and domain)

[EricGuic]

Hi, anyone to give me some advice ? In your opinion, is this a known issue, or more likely a misconfiguration (maybe in the database) ? I've seen the post about the v3 developpment : should I understand that this problem won't be adress and I have to wait for the v3 release ? Are other people using also the yobasystem databse instead of the jc21 one ? (as already mentionned, I've followed a guide online for my Raspberry pi). Do you think that the jc21 databse will fix this ? Thanks for your advice.

[Kirk1984]

The last console outputs states that your certs are still valid so they are skipped for renewal.

On the database topic: you don't need it really. npm 3 will switch to an internal sqlite (as far as i unterstand). i used the official mariadb initially but i tossed it and i use the sqlite database. runs fine.

EDIT: I think the jc21 database is only there because he added the aria storage engine. but that is standard in mariadb since 10.4. So i guess that maybe that database might vanish :)

[wtrdk]

I experience a similar issue. An advise from another user was to delete the certificate and request a new one, but it's not working. For some subdomains renewing worked fine, for some it failed and requesting a new one worked fine. And for some both options didn't work. Below a part of the logfile for requesting a new certificate after deleting the expired one through the webinterface.

Failed to renew certificate npm-8 with error: Some challenges have failed.

All renewals failed. The following certificates could not be renewed:

  /etc/letsencrypt/live/npm-8/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

    at ChildProcess.exithandler (node:child_process:397:12)

    at ChildProcess.emit (node:events:394:28)

    at maybeClose (node:internal/child_process:1064:16)

    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)

Connection Error: Error: read ECONNRESET

Connection Error: Error: read ECONNRESET

[9/7/2021] [6:41:04 AM] [Express  ] › ⚠  warning   Command failed: /usr/sbin/nginx -t -g "error_log off;"

nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-1/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/npm-1/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

nginx: configuration file /etc/nginx/nginx.conf test failed

[EricGuic]

Hi, thanks both of you for your replies. Today I've tried another way, by requesting a wildcart certificate through a DNS Challenge. My domain name provider is OVH, I've correctly generate and copy/paste the key/secrets in NPM fields :

dns_ovh_endpoint = ovh-eu
dns_ovh_application_key = **********
dns_ovh_application_secret = **********
dns_ovh_consumer_key = **********

(I've of course replaced it by stars ;-)

And after a few minutes, here is the error message NPM gives me :

Error: Command failed: pip install certbot-dns-ovh==1.8.0 
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/certbot-dns-ovh/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/certbot-dns-ovh/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/certbot-dns-ovh/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/certbot-dns-ovh/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/certbot-dns-ovh/
ERROR: Could not find a version that satisfies the requirement certbot-dns-ovh==1.8.0 (from versions: none)
ERROR: No matching distribution found for certbot-dns-ovh==1.8.0

    at ChildProcess.exithandler (node:child_process:397:12)
    at ChildProcess.emit (node:events:394:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)

Do you find any clues in this error message ?

Thanks per advance. Eric

[thebiblelover7]

@EricGuic Seems like your DNS isn't working well as you can see in the Temporary failure in name resolution

Are you sure you have pointed your DNS to the correct IP?

Edit: On second reading I noticed that it is the command to install certbot that is failing. This means that your container/npm cannot get an IP address from which to install certbot. Check https://stackoverflow.com/a/46629043/16625037 and see if that solves your problem. Let me know how it goes!

[EricGuic]

Check https://stackoverflow.com/a/46629043/16625037 and see if that solves your problem. Let me know how it goes!

@thebiblelover7 , thanks to take time to read the log, I will check your link.

[EricGuic]

@thebiblelover7 I think I’m maybe not good enough to fully understand your link. But from the first day I installed this raspberry, I was wondering on how to correctly setup the network part.

As I said, this raspberry run Adguard Home in a docker container on bridge network, and the pi has the 192.168.1.10 ip adress. The router is set up with this .10 ip adress as a dns provider. Nginx PM is in another container, also on bridge network. Do I have to set somewhere in the NPM container to use the DNS of the host, or another DNS set manually (like 8.8.8.8) ? Maybe my Adguard Home is blocking some access to certbot or to Let’sEncrypt ? (but I’ve never find online a list of ip or domains used by LE for the certification process).

[thebiblelover7]

@thebiblelover7 I think I’m maybe not good enough to fully understand your link. But from the first day I installed this raspberry, I was wondering on how to correctly setup the network part.

As I said, this raspberry run Adguard Home in a docker container on bridge network, and the pi has the 192.168.1.10 ip adress. The router is set up with this .10 ip adress as a dns provider. Nginx PM is in another container, also on bridge network. Do I have to set somewhere in the NPM container to use the DNS of the host, or another DNS set manually (like 8.8.8.8) ? Maybe my Adguard Home is blocking some access to certbot or to Let’sEncrypt ? (but I’ve never find online a list of ip or domains used by LE for the certification process).

@EricGuic Let me try to simplfy this:

Just watch the video below and afterwards reboot

https://user-images.githubusercontent.com/61815862/132990877-92072d0e-d2e6-4bd0-93b8-5d8220012ebc.mp4

I hope this makes sense

[EricGuic]

Hi @thebiblelover7 , thank you very much for taking the time to do this little video. I've followed your guide, and modified the DNS like you show me (primary the raspberry itself with Adguard home, and as a "plan B" the Cloudflare DNS).

I've also run this command to change the default settings of the pi : sudo nano /etc/dhcpcd.conf I've set a fallback to a static ip (192.168.1.10) and I set this two DNS again.

I've update+upgrade the pi, reboot it, but I still have the same error in the log of the container : (I've only copied the relevant lines IMHO)

[9/12/2021] [4:29:30 PM] [IP Ranges] › ✖  error     getaddrinfo EAI_AGAIN ip-ranges.amazonaws.com,
[9/12/2021] [4:29:44 PM] [Express  ] › ⚠  warning   invalid signature,
[9/12/2021] [4:30:38 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #37: home.mydomain.fr,
[9/12/2021] [4:30:38 PM] [SSL      ] › ℹ  info      Command: certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-37" --agree-tos --authenticator webroot --email "email@mydomain.fr" --preferred-challenges "dns,http" --domains "home.mydomain.fr" ,
[9/12/2021] [4:30:50 PM] [Express  ] › ⚠  warning   Command failed: certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-37" --agree-tos --authenticator webroot --email "email@mydomain.fr" --preferred-challenges "dns,http" --domains "home.mydomain.fr" ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
An unexpected error occurred:,
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xb5a5ad70>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')),
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Any idea ? Should I try to completely erase and re-deploy my NPM container ? (with the mariaDB one) ?

(If that can help, I've also tried to request a wildcard certificate using the DNS Challenge method, by providing the token from my provider OVH). No way to make it work neither.)

[thebiblelover7]

@EricGuic Check this video, hope it helps!

https://user-images.githubusercontent.com/61815862/133076566-b7e92e93-a364-4091-867b-75e1a2981540.mp4

[EricGuic]

Hi @thebiblelover7 , thanks again for your tip, it force me to go back to the basics (which I did'nt master apparently). So I was only getting ping respond for 8.8.8.8 but not from google.com. After settings the DNS of the host, I've search how to force the DNS of the docker container and found that guide online : https://robinwinslow.uk/fix-docker-networking-dns

You can set the default DNS settings options for the docker daemon by creating a daemon configuration file at /etc/docker/daemon.json.

{
    "dns": ["1.1.1.1", "8.8.8.8"]
}

I've choosed to completely "overpassed" my AdGuardHome DNS (running on another container on the same raspberry pi) to see if it could be the problem (so Cloudflare as primary, and Google as secondary DNS) And with that settings (and maybe in addition with the previous one made on /etc/dhcpcd.conf), everything is now working as expected. Hourra !

I've successfully setup a wildcard certificate for my domain (with the DNS challenge). A've also tested LinuxServer SWAG, and it was giving me the same error until I set the daemon.json files.

So thank you again for your help :-)

(may I try a last thing : what is your advice about the network config : with only AdGuard Home, watchtower (mode notify only) and NPM on the Pi, should I run NPM under the bridge network or under the specific network created by Portainer when I deployed the stack ? (see capture attached of my current config).

[S1M8N]

Hello,

I rewrite into this issue because I don't have inter connection with my docker and my server and into my portainer network list, I don't have brige system :

Do you have an idea how can I import the brige system configuration ?

Thank you in advenced

[thebiblelover7]

Hello,

I rewrite into this issue because I don't have inter connection with my docker and my server and into my portainer network list, I don't have brige system :

Do you have an idea how can I import the brige system configuration ?

Thank you in advenced

@S1M8N This is not a nginx-proxy-manager issue, please pass this on to portainer/docker