search

home-assistant / addons

:heavy_plus_sign: Docker add-ons for Home Assistant
https://home-assistant.io/hassio/
Apache License 2.0
1.48k stars 1.45k forks source link

Letsencrypt with local CA not working #2713

Closed Eugeniusz-Gienek closed 1 year ago

Eugeniusz-Gienek commented 1 year ago

Describe the issue you are experiencing

Dear team, There is an issue with the local CAs. The local CA is usually signed by... this CA, thus the SSL certificate is treated by the SSL library initially as self-signed - e.g. not a trusted one. This leads to inability to setup the plugin with local CA - and yes, I have filled the certificate section with the correct root certificate. The plugin log looks like this:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/file-structure.sh
cont-init: info: /etc/cont-init.d/file-structure.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun lets-encrypt (no readiness notification)
s6-rc: info: service legacy-services successfully started
[00:26:11] INFO: Selected http verification
Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
requests.exceptions.SSLError: HTTPSConnectionPool(host='myca.local', port=8443): Max retries exceeded with url: /acme/acme/directory (Caused by SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)')))
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

after that the plugin immediately dies. I have even tried to do the following - pass a certificate itself via /ssl folder and set the global variable REQUESTS_CA_BUNDLE to point to it in the scripts /etc/cont-init.d/file-structure.sh and /etc/services.d/lets-encrypt/run - no luck (however that seemed to work in another container - homeassistant one - when I've installed there via apk the certbot)

I have even managed to get the lets-encrypt log (domains replaced):

bash-5.1# cat /var/log/letsencrypt/letsencrypt.log
2022-10-20 00:45:27,954:DEBUG:certbot._internal.main:certbot version: 1.21.0
2022-10-20 00:45:27,955:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/local/bin/certbot
2022-10-20 00:45:27,955:DEBUG:certbot._internal.main:Arguments: ['--non-interactive', '--keep-until-expiring', '--expand', '--email', ‚user@host.com', '--agree-tos', '--config-dir', '/data/letsencrypt', '--work-dir', '/data/workdir', '--preferred-challenges', 'http', '-d', 'homeassistant.local', '--server', 'https://myca.local:8443/acme/acme/directory', '--standalone', '--preferred-chain', 'ISRG Root X1']
2022-10-20 00:45:27,955:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#certbot-dns-azure:dns-azure,PluginEntryPoint#certbot-dns-directadmin:directadmin,PluginEntryPoint#certbot-dns-hetzner:dns-hetzner,PluginEntryPoint#certbot-dns-netcup:dns-netcup,PluginEntryPoint#certbot-dns-njalla:dns-njalla,PluginEntryPoint#certbot-dns-transip:dns-transip,PluginEntryPoint#certbot-plugin-gandi:dns,PluginEntryPoint#certbot-plugin-gandi:dns-gandi,PluginEntryPoint#certbot-route53:auth,PluginEntryPoint#directadmin,PluginEntryPoint#dns,PluginEntryPoint#dns-azure,PluginEntryPoint#dns-cloudflare,PluginEntryPoint#dns-cloudxns,PluginEntryPoint#dns-digitalocean,PluginEntryPoint#dns-dnsimple,PluginEntryPoint#dns-dnsmadeeasy,PluginEntryPoint#dns-gandi,PluginEntryPoint#dns-gehirn,PluginEntryPoint#dns-google,PluginEntryPoint#dns-hetzner,PluginEntryPoint#dns-linode,PluginEntryPoint#dns-luadns,PluginEntryPoint#dns-netcup,PluginEntryPoint#dns-njalla,PluginEntryPoint#dns-nsone,PluginEntryPoint#dns-ovh,PluginEntryPoint#dns-rfc2136,PluginEntryPoint#dns-route53,PluginEntryPoint#dns-sakuracloud,PluginEntryPoint#dns-transip,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-10-20 00:45:27,975:DEBUG:certbot._internal.log:Root logging level set at 30
2022-10-20 00:45:27,976:DEBUG:certbot._internal.plugins.selection:Requested authenticator standalone and installer None
2022-10-20 00:45:27,978:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: Authenticator, Plugin
Entry point: standalone = certbot._internal.plugins.standalone:Authenticator
Initialized: <certbot._internal.plugins.standalone.Authenticator object at 0x7f62016d37c0>
Prep: True
2022-10-20 00:45:27,978:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.standalone.Authenticator object at 0x7f62016d37c0> and installer None
2022-10-20 00:45:27,979:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator standalone, Installer None
2022-10-20 00:45:28,099:DEBUG:acme.client:Sending GET request to https://myca.local:8443/acme/acme/directory.
2022-10-20 00:45:28,102:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): myca.local:8443
2022-10-20 00:45:28,107:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 402, in ssl_wrap_socket
    context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data)
ssl.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 404, in ssl_wrap_socket
    raise SSLError(e)
urllib3.exceptions.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=‚myca.local’, port=8443): Max retries exceeded with url: /acme/acme/directory (Caused by SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/local/lib/python3.9/site-packages/certbot/_internal/main.py", line 1574, in main
    return config.func(config, plugins)
  File "/usr/local/lib/python3.9/site-packages/certbot/_internal/main.py", line 1416, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/usr/local/lib/python3.9/site-packages/certbot/_internal/main.py", line 763, in _init_le_client
    acc, acme = _determine_account(config)
  File "/usr/local/lib/python3.9/site-packages/certbot/_internal/main.py", line 680, in _determine_account
    acc, acme = client.register(
  File "/usr/local/lib/python3.9/site-packages/certbot/_internal/client.py", line 184, in register
    acme = acme_from_config_key(config, key)
  File "/usr/local/lib/python3.9/site-packages/certbot/_internal/client.py", line 48, in acme_from_config_key
    client = acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/usr/local/lib/python3.9/site-packages/acme/client.py", line 880, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/usr/local/lib/python3.9/site-packages/acme/client.py", line 1242, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/local/lib/python3.9/site-packages/acme/client.py", line 1180, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='myca.local’, port=8443): Max retries exceeded with url: /acme/acme/directory (Caused by SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)')))
2022-10-20 00:45:28,110:ERROR:certbot._internal.log:An unexpected error occurred:
2022-10-20 00:45:28,110:ERROR:certbot._internal.log:requests.exceptions.SSLError: HTTPSConnectionPool(host='myca.local’, port=8443): Max retries exceeded with url: /acme/acme/directory (Caused by SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)')))
bash-5.1# #

Yaml config:

domains:
  - homeassistant.local
email: user@host.com
keyfile: privkey.pem
certfile: fullchain.pem
challenge: http
dns: {}
acme_server: https://myca.local:8443/acme/acme/directory
acme_root_ca_cert: >-
  -----BEGIN CERTIFICATE-----
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus nec rutrum lectus, sed sagittis lectus. Donec vel condimentum nulla. Phasellus varius fringilla tortor, vitae congue sem tempus non. Etiam nec leo feugiat, ultrices elit vel, ultrices eros. Vivamus vitae ex gravida, fringilla augue vel, interdum sapien. Aliquam erat volutpat. Interdum et malesuada fames ac ante ipsum primis in faucibus. 
  -----END CERTIFICATE-----

It would be nice if that could be fixed. Or maybe I am doing smth wrong?

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Which add-on are you reporting an issue with?

Let's Encrypt

What is the version of the add-on?

4.12.7

Steps to reproduce the issue

  1. Install the plugin
  2. Configure the plugin, including the CA certificate, for the local CA
  3. Run the plugin
  4. Get an error

System Health information

Currently there is nothing to repair (it is in Polish: "Obecnie nie ma nic do naprawy")

Anything in the Supervisor logs that might be useful for us?

Logger: homeassistant.components.hassio
Source: components/hassio/websocket_api.py:123
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 00:23:30 (6 occurrences)
Last logged: 00:55:49
Failed to to call /addons/core_letsencrypt/stats - Container addon_core_letsencrypt is not running

Anything in the add-on logs that might be useful for us?

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/file-structure.sh
cont-init: info: /etc/cont-init.d/file-structure.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun lets-encrypt (no readiness notification)
s6-rc: info: service legacy-services successfully started
[00:45:26] INFO: Selected http verification
Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
requests.exceptions.SSLError: HTTPSConnectionPool(host='myca.local', port=8443): Max retries exceeded with url: /acme/acme/directory (Caused by SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)')))
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Additional information

No response

mrgohin commented 1 year ago

Confirmed. My guess is that lets encrypt is implemented as container and is not using the configured dns from hass os.

I think I was correct. After I used following command: ha dns options --servers dns://.... --servers dns:// to set my internal server the challange was successful.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Eugeniusz-Gienek commented 1 year ago

So does the above command fix the issue and it is not an issue or should it be fixed somehow?

mrgohin commented 1 year ago

I don't think so since the addon doesn't work without setting non default dns options in Hass OS.

Also I struggle with the fact that the addon is not renewing my certificate automatically and requires a manual reboot of my whole kvm instance and firing following command by hand afterwards: addons start core_letsencrypt

robby-d commented 1 year ago

I can confirm this issue as well. In my case I already have hass OS's DNS servers set to my internal DNS server (I tried that ha dns options command at the hass terminal and it didn't fix the issue.. or does the DNS need to be changed at the level of the letsencrypt docker container, @m4k5ym ?

mrgohin commented 1 year ago

Nope I just used that command than at least it worked but still the renewal is failing. I think that add-on should be reworked because in my opinion it is not meeting the quality requirements of hass.

miguelangel-nubla commented 1 year ago

I also encountered this, it is a YAML formatting issue. The solution is to edit the addon config in YAML mode (three dots on the top right of the options -> Edit in YAML), and instead of using a '>-' block for acme_root_ca_cert:

acme_root_ca_cert: >-
  -----BEGIN CERTIFICATE-----
  3bn5k23bn5kjKGGhjk32grk
  3bn5k23bn5kjKGGhjk32grk
  3bn5k23bn5kjKGGhjk32grk
  3bn5k -----END CERTIFICATE-----

use a pipe symbol '|' block:

acme_root_ca_cert: |
  -----BEGIN CERTIFICATE-----
  3bn5k23bn5kjKGGhjk32grk
  3bn5k23bn5kjKGGhjk32grk
  3bn5k23bn5kjKGGhjk32grk
  3bn5k
  -----END CERTIFICATE-----

to preserve formatting and newlines intact, mind the two space indentation to the certificate.

Then the ca file is correctly created internally and certbot is able to read the certificates from it.

Eugeniusz-Gienek commented 1 year ago

I also encountered this, it is a YAML formatting issue. The solution is to edit the addon config in YAML mode (three dots on the top right of the options -> Edit in YAML), and instead of using a '>-' block for acme_root_ca_cert:

acme_root_ca_cert: >-
  -----BEGIN CERTIFICATE-----
  3bn5k23bn5kjKGGhjk32grk
  3bn5k23bn5kjKGGhjk32grk
  3bn5k23bn5kjKGGhjk32grk
  3bn5k -----END CERTIFICATE-----

use a pipe symbol '|' block:

acme_root_ca_cert: |
  -----BEGIN CERTIFICATE-----
  3bn5k23bn5kjKGGhjk32grk
  3bn5k23bn5kjKGGhjk32grk
  3bn5k23bn5kjKGGhjk32grk
  3bn5k
  -----END CERTIFICATE-----

to preserve formatting and newlines intact, mind the two space indentation to the certificate.

Then the ca file is correctly created internally and certbot is able to read the certificates from it.

Thank You - that worked, finally! Awesome, You are my saviour!