Let's Encrypt: Internal Error

[TheZoker]

I used the hassio addon for quite some time now. Today I tried to migrate the proxy manager to my synology. For that I use the synology docker manager and this docker image.

So I installed the latest version (1.7.0) and configured my port forwarding. The only issue I had was with the lets encrypt certificate. When I tried to request a certificate, I got an "Internal error":

This is the content in the /config/log/letsencrypt/letsencrypt.log:

2020-03-18 17:02:07,145:DEBUG:certbot.main:certbot version: 0.30.2
2020-03-18 17:02:07,147:DEBUG:certbot.main:Arguments: ['--non-interactive', '--config', '/etc/letsencrypt.ini', '--cert-name', 'npm-15', '--agree-tos', '--email', 'mail@zoker.me', '--preferred-challenges', 'dns,http', '--webroot', '--domains', 'wg.zkr.io']
2020-03-18 17:02:07,150:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-03-18 17:02:07,202:DEBUG:certbot.log:Root logging level set at 20
2020-03-18 17:02:07,204:INFO:certbot.log:Saving debug log to /config/log/letsencrypt/letsencrypt.log
2020-03-18 17:02:07,206:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
2020-03-18 17:02:07,220:DEBUG:certbot.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7fded3100a90>
Prep: True
2020-03-18 17:02:07,223:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7fded3100a90> and installer None
2020-03-18 17:02:07,224:INFO:certbot.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2020-03-18 17:02:07,320:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2020-03-18 17:02:07,326:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2020-03-18 17:02:12,333:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fded3046d30>: Failed to establish a new connection: [Errno -3] Try again

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fded3046d30>: Failed to establish a new connection: [Errno -3] Try again',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.30.2', 'console_scripts', 'certbot')()
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 1233, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 604, in _init_le_client
    acc, acme = _determine_account(config)
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 521, in _determine_account
    config, account_storage, tos_cb=_tos_cb)
  File "/usr/lib/python3.6/site-packages/certbot/client.py", line 181, in register
    acme = acme_from_config_key(config, key)
  File "/usr/lib/python3.6/site-packages/certbot/client.py", line 51, in acme_from_config_key
    return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/usr/lib/python3.6/site-packages/acme/client.py", line 814, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/usr/lib/python3.6/site-packages/acme/client.py", line 1152, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/lib/python3.6/site-packages/acme/client.py", line 1101, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 513, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fded3046d30>: Failed to establish a new connection: [Errno -3] Try again',))
2020-03-18 17:02:12,340:ERROR:certbot.log:An unexpected error occurred:

Is this an issue with this image or with the base image or is my configuration wrong?

Is this issue related? https://github.com/jc21/nginx-proxy-manager/issues/180#issuecomment-599278001

Thank you!

Edit: A little bit more information: I also tried version 1.6.0, but it has the same issue. For both version: Connecting to my devices trough the manager using port 80 works fine. Connecting to them via 443 leads to a empty response.

[jlesage]

Looks like the container is not able to reach the LetEncrypt server.

Can you try:

docker exec <container name> ping acme-v02.api.letsencrypt.org

[TheZoker]

Hmm yes, something does not seem right:

When I try the same with my AdGuard container it works:

AdGuard is running in host mode, while ProxyManager is running in bridge mode. But bridge mode should work as well, right?

Do you have any idea, how I can fix that?

And thank you very much for your help! Much appreciated

Edit: In my AdGuard log I can see, that the request from the proxy manager was successful, but somehow the command still fails:

[jlesage]

Are you using AdGuard as your DNS server ? If yes. this may be the cause of the issue.

Are you able to ping acme-v02.api.letsencrypt.org from your Synology ?

[TheZoker]

Yes I'm using it as my DNS server.

Yes the ping from synology itself works:

[jlesage]

Is this DNS server also used by you Synology? You can try to run the following command on your Synology: nslookup acme-v02.api.letsencrypt.org <DNS server IP>.

Also, can you confirm that this DNS server used by the container:

docker exec <container name> cat /etc/resolv.conf

[bookandrelease]

I am having the same internal error message. I use dnsimple and I do have a certificate on my account for my domain. Pinging the address you said above is successful.

[jlesage]

If your container accessible from the internet on port 80 ? You can check with https://www.yougetsignal.com/tools/open-ports/

[mdisieno]

Same issue here. I am able to ping with: docker exec ping acme-v02.api.letsencrypt.org No internal DNS beyond a redirect to an external PiHole server. I tried both requesting a new cert (internal error) and creating my own via a LetsEncrypt container. Both no dice.

[jlesage]

Make sure the container is reachable from the internet on port 80. You can check with https://www.yougetsignal.com/tools/open-ports/

[mdisieno]

Morning. Port shows as not reachable but both my router settings and the use of the proxy manager is present over port 80. If I use the IP:80 I go directly to where I set redirect to.

[mdisieno]

443 shows open and listening. Could I just direct through that? 80 is still working fine, It appears my isp blocks port 80, though I can go it via both directly through the IP and via my duckdns dyndns.

[jlesage]

If your ISP blocks port 80, then I guess your cannot use Nginx Proxy Manager with automatic certificate generation from LetsEncrypt. Port 80 is required. This is how LetsEncrypt validates that your are the owner of the DNS name.

[mdisieno]

Odd given I can use port 80 for the hosts from Nginx Proxy Manager. I'll do some more research and report back on a fix for future reference.