Let'sEncrypt: dns-he: unable to set up DNS validation of subdomain

[eegerferenc]

Describe the issue you are experiencing

In the current implementation of Hurricane Electric DNS plugin of Let'sEncrypt, there is no option for performing domain validation of subdomain. E.g. for the managed domain "eegerferenc.org", only "eegerferenc.org" or "*.eegerferenc.org" can be validated, but not for example "homeassistant.eegerferenc.org".

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Which add-on are you reporting an issue with?

Let's Encrypt

What is the version of the add-on?

5.0.18

Steps to reproduce the issue

1. Have a managed domain with a subdomain at dns.he.net registered

2. Set up Let'sEncrypt accordingly:

3. Start Let'sEncrypt to generate/renew certificate ...

System Health information

System Information

version	core-2024.6.1
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.12.2
os_name	Linux
os_version	6.6.28-haos-raspi
arch	armv7l
timezone	Europe/Budapest
config_dir	/config

Home Assistant Cloud

logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok

Home Assistant Supervisor

host_os | Home Assistant OS 12.3 -- | -- update_channel | stable supervisor_version | supervisor-2024.06.0 agent_version | 1.6.0 docker_version | 25.0.5 disk_total | 28.0 GB disk_used | 6.5 GB healthy | true supported | true host_connectivity | true supervisor_connectivity | true ntp_synchronized | true virtualization | board | rpi3 supervisor_api | ok version_api | ok installed_addons | NGINX Home Assistant SSL proxy (3.9.0), File editor (5.8.0), Let's Encrypt (5.0.18), ESPHome (2024.5.5), Mosquitto broker (6.4.1), Terminal & SSH (9.14.0)

Dashboards

dashboards | 3 -- | -- resources | 0 views | 4 mode | storage

Recorder

oldest_recorder_run | May 14, 2024 at 00:27 -- | -- current_recorder_run | June 10, 2024 at 13:08 estimated_db_size | 574.46 MiB database_engine | sqlite database_version | 3.44.2

Anything in the Supervisor logs that might be useful for us?

No response

Anything in the add-on logs that might be useful for us?

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/file-structure.sh
cont-init: info: /etc/cont-init.d/file-structure.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun lets-encrypt (no readiness notification)
s6-rc: info: service legacy-services successfully started
[14:07:32] INFO: Selected DNS Provider: dns-he
[14:07:33] INFO: Use propagation seconds: 300
[14:07:33] INFO: Detecting existing certificate type for homeassistant.eegerferenc.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for homeassistant.eegerferenc.org
Encountered exception during recovery: certbot.errors.PluginError: Unable to find domain: homeassistant.eegerferenc.org
Unable to find domain: homeassistant.eegerferenc.org
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Additional information

No response

[eegerferenc]

Meanwhile, I dug into the problem in detail. What I found is the following:

First, a bit of history:

• Back in the late 2010s, when Lets Encrypt and ACME were introduced, HE had no such feature as "dynamic TXT record". The only way to add a TXT was to log into their website and add it manually. The feature was asked for, but they didn't implemented it instantly.
• Meanwhile, the community, or more precisely, tsaaristo did not wait for them and in 2019 made a makeshift "client" for HE, which actually simulated a flesh-blood user's activity on the web interface (as no API was available). This "client" was uploaded to Github and then to PyPI as "certbot-dns-he 1.0.0". However, it has a serious flaw: it can only work as long as the name of the domain to be updated matches the DNS zone name, that is, it works only for the root domain within the zone.
• Sometime in 2020, HE implemented their dynamic TXT feature, allowing the provisioning of TXT records via an API. By that time, certbot already used the certbot-dns-he plugin, so the new API went ignored. That plugin was forked several times on Github, with the problematic part (handling of the case where zone and domain name is different) rectified.

The situation from HA's perspective is rather problematic because of the following:

• The original "certbot-dns-he 1.0.0" of tsaaristo at Github is a dead repo: it was not touched since its initial commit in 2019. It has open issues and pull requests with the needed fix... unanswered and unmerged since 2020.
• The "certbot-dns-he 1.0.0" package on PyPI (also (un)authored by tsaaristo) is also stale since 2019.
• On "ordinary" systems, this is not a problem: as long as one needs just dynamic DNS (the primary benefit of HE) with TLS without subdomains (the usual use-case) that's OK out-of-the-box. If one indeed does need subdomains, one may simply use a patched version of the certbot-dns-he plugin from Github.
• On Home Assistant, as long as your instance is offline or online via HA Cloud or online in your root domain, that's OK. But if you want to put your HA under a subdomain (e.g. you have other TLS-using servers behind the same public IP and you want neither to have the private key shared across multiple systems nor have multiple valid certs for the same domain out there) then you have the problem: since the add-on is isolated into a Docker container and has its own way of pulling its dependencies (the original certbot-dns-he 1.0.0 among them), you cannot simply replace the flawed PyPI version with another Github fork.

Currently, I see the following possiblity of solving the problem. First, it has to be ensured that a correctly-functioning version of certbot-dns-he is available on PyPI (as HA add-ons pull their dependencies from there). I try to reach out to tsaaristo and ask him politely to merge the stalled PR from 2020. If this fails, one of the forked versions needs to be added to PyPI. Second, in any case, the dependency listing in LetsEncrypt add-on has to be updated to pull-in the corrected version.