search

NginxProxyManager / nginx-proxy-manager

Docker container for managing Nginx proxy hosts with a simple, powerful interface
https://nginxproxymanager.com
MIT License
22.89k stars 2.66k forks source link

NPM docker - Network unreachable when connecting to acme-v02.api.letsencrypt.org #3954

Open matteo-martinelli opened 2 months ago

matteo-martinelli commented 2 months ago

Checklist

Describe the bug

Whenever activity I try to take involving SSL certificates, fails. The recurrent error is: Requesting acme-v02.api.letsencrypt.org/directory: Network is unreachable The same error appears both for automatic renewals, as well as for new manual renewals.

The logs are saved in /tmp/letsencrypt-log/letsencrypt.log, the content of one of the several logs here contained is the following: 

2024-08-24 13:32:37,832:DEBUG:certbot._internal.main:certbot version: 2.11.0
2024-08-24 13:32:37,833:DEBUG:certbot._internal.main:Location of certbot entry point: /opt/certbot/bin/certbot
2024-08-24 13:32:37,833:DEBUG:certbot._internal.main:Arguments: ['--force-renewal', '--config', '/etc/letsencrypt.ini', '--work-dir', '/tmp/letsencrypt-lib', '--logs-dir', '/tmp/letsencrypt-log', '--cert-name', 'npm-7', '--preferred-challenges', 'dns,http', '--no-random-sleep-on-renew', '--disable-hook-validation']
2024-08-24 13:32:37,833:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2024-08-24 13:32:37,893:DEBUG:certbot._internal.log:Root logging level set at 30
2024-08-24 13:32:37,895:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/npm-7.conf
2024-08-24 13:32:37,898:DEBUG:certbot.configuration:Var pref_challs=['dns-01', 'http-01'] (set by user).
2024-08-24 13:32:37,898:DEBUG:certbot.configuration:Var logs_dir=/tmp/letsencrypt-log (set by user).
2024-08-24 13:32:37,898:DEBUG:certbot.configuration:Var work_dir=/tmp/letsencrypt-lib (set by user).
2024-08-24 13:32:37,899:DEBUG:certbot._internal.plugins.selection:Requested authenticator None and installer None
2024-08-24 13:32:37,899:DEBUG:certbot.configuration:Var preferred_chain=ISRG Root X1 (set by user).
2024-08-24 13:32:37,899:DEBUG:certbot.configuration:Var key_type=ecdsa (set by user).
2024-08-24 13:32:37,900:DEBUG:certbot.configuration:Var elliptic_curve=secp384r1 (set by user).
2024-08-24 13:32:37,900:DEBUG:certbot.configuration:Var webroot_path=['/data/letsencrypt-acme-challenge'] (set by user).
2024-08-24 13:32:37,900:DEBUG:certbot.configuration:Var webroot_map={'webroot_path'} (set by user).
2024-08-24 13:32:37,900:DEBUG:certbot.configuration:Var webroot_path=['/data/letsencrypt-acme-challenge'] (set by user).
2024-08-24 13:32:37,945:DEBUG:certbot._internal.renewal:Auto-renewal forced with --force-renewal...
2024-08-24 13:32:37,946:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2024-08-24 13:32:37,946:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * webroot
Description: Saves the necessary validation files to a .well-known/acme-challenge/ directory within the nominated webroot path. A seperate HTTP server must be running and serving files from the webroot path. HTTP challenge only (wildcards not supported).
Interfaces: Authenticator, Plugin
Entry point: EntryPoint(name='webroot', value='certbot._internal.plugins.webroot:Authenticator', group='certbot.plugins')
Initialized: <certbot._internal.plugins.webroot.Authenticator object at 0x7f782f10e650>
Prep: True
2024-08-24 13:32:37,947:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.webroot.Authenticator object at 0x7f782f10e650> and installer None
2024-08-24 13:32:37,947:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2024-08-24 13:32:38,273:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/1136533627', new_authzr_uri=None, terms_of_service=None), ce3f19a8ed7c5133f98fbd0d8451ab78, Meta(creation_dt=datetime.datetime(2023, 5, 31, 13, 52, 57, tzinfo=<UTC>), creation_host='mex-X520', register_to_eff=None))>
2024-08-24 13:32:38,274:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2024-08-24 13:32:38,278:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2024-08-24 13:33:23,363:ERROR:certbot._internal.renewal:Failed to renew certificate npm-7 with error: Requesting acme-v02.api.letsencrypt.org/directory: Network is unreachable
2024-08-24 13:33:23,385:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/connection.py", line 616, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f782d389110>: Failed to establish a new connection: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/certbot/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f782d389110>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/certbot/lib/python3.11/site-packages/acme/client.py", line 647, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f782d389110>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/certbot/lib/python3.11/site-packages/certbot/_internal/renewal.py", line 540, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/opt/certbot/lib/python3.11/site-packages/certbot/_internal/main.py", line 1548, in renew_cert
    le_client = _init_le_client(config, auth, installer)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/certbot/_internal/main.py", line 838, in _init_le_client
    return client.Client(config, acc, authenticator, installer, acme=acme)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/certbot/_internal/client.py", line 297, in __init__
    acme = acme_from_config_key(config, self.account.key, self.account.regr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/certbot/_internal/client.py", line 72, in acme_from_config_key
    directory = acme_client.ClientV2.get_directory(config.server, net)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/acme/client.py", line 330, in get_directory
    return messages.Directory.from_json(net.get(url).json())
                                        ^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/acme/client.py", line 705, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/acme/client.py", line 669, in _send_request
    raise ValueError(f"Requesting {host}{path}:{err_msg}")
ValueError: Requesting acme-v02.api.letsencrypt.org/directory: Network is unreachable

2024-08-24 13:33:23,403:DEBUG:certbot._internal.display.obj:Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2024-08-24 13:33:23,403:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2024-08-24 13:33:23,404:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/npm-7/fullchain.pem (failure)
2024-08-24 13:33:23,404:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2024-08-24 13:33:23,408:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/opt/certbot/bin/certbot", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/certbot/_internal/main.py", line 1894, in main
    return config.func(config, plugins)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/certbot/_internal/main.py", line 1642, in renew
    renewed_domains, failed_domains = renewal.handle_renewal_request(config)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/lib/python3.11/site-packages/certbot/_internal/renewal.py", line 568, in handle_renewal_request
    raise errors.Error(
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2024-08-24 13:33:23,419:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)

The error "network is unreachable" also appears when trying to issue a new certificate from the UI: 

CommandError: Saving debug log to /tmp/letsencrypt-log/letsencrypt.log
An unexpected error occurred:
ValueError: Requesting acme-v02.api.letsencrypt.org/directory: Network is unreachable
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/letsencrypt-log/letsencrypt.log or re-run Certbot with -v for more details.

    at /app/lib/utils.js:16:13
    at ChildProcess.exithandler (node:child_process:430:5)
    at ChildProcess.emit (node:events:519:28)
    at maybeClose (node:internal/child_process:1105:16)
    at ChildProcess._handle.onexit (node:internal/child_process:305:5)

Nginx Proxy Manager Version

2.11.3

To Reproduce Steps to reproduce the behavior:

Expected behavior

A correct generation or renewal of an SSL certificate, without errors. The same for automatic renewals, which suffer of the same problem.

Screenshots

Operating System

Ubuntu 22.04.4 LTS, Docker 27.1.2, NPM running on docker-compose: 

version: '3.8'

services:
  app:
    # Releases page: https://github.com/NginxProxyManager/nginx-proxy-manager/releases
    image: jc21/nginx-proxy-manager:2.11.3
    container_name: Nginx-Proxy-Manager
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    volumes:
      - ./nginx-container/data:/data
      - ./nginx-container/letsencrypt:/etc/letsencrypt
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    environment:
      - TZ=Europe/Rome

Network used is the default generated by the compose, 

[
    {
        "Name": "nginx-proxy_default",
        "Id": "...",
        "Created": "2024-08-24T13:32:27.493162071+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "b18e610d557d3db0dd6e5fd54ecfb0ccb8864515a1cb20c4aa8f739c17246600": {
                "Name": "Nginx-Proxy-Manager",
                "EndpointID": "...",
                "MacAddress": "...",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "nginx-proxy",
            "com.docker.compose.version": "1.29.2"
        }
    }
]

Additional context

My domain is provided by SquareSpace (previously Google Domain, then migrated).

When doing traceroute acme-v02.api.letsencrypt.org from inside the container, i get: 

traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  172.20.0.1 (172.20.0.1)  0.107 ms  0.064 ms  0.061 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

When using ping from inside the container, the console returns: PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data., then blocks.

As you can see from traceroute and ping, the domain is correctly resolved, therefore I exclude any DNS problem.

When instead using traceroute from the host machine, i get: 

traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  mex-X520 (172.65.0.1)  3062.524 ms !H  3062.421 ms !H  3062.364 ms !H

When using ping from the host machine, the console returns: 

PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data.
From mex-X520 (172.65.0.1) icmp_seq=1 Destination Host Unreachable
From mex-X520 (172.65.0.1) icmp_seq=2 Destination Host Unreachable
...

From other machines, instead, also from withing the same network, acme-v02.api.letsencrypt.org is normally reachable I really can't figure out what's going on both under the docker container and the host machine as well.

raenrfm commented 1 month ago

Experiencing the same bug as well.