LetsEncrypt not auto renewing certificates

[borpin]

Supervised Install.

I have just had a renewal notice for the certificate for the domain name I use for HA and setup using the LetsEncrypt Plugin. On the basis of getting the email, the renewal should have already happened (certbot does not wait that late).

It appears the auto-renew is not working.

When the addon is manually started, the renewal occurs.

letsencrypt log showing previous auto renewal attempt plus the renewal done when the addon was restarted.

2020-07-08 01:50:16,689:DEBUG:certbot.main:certbot version: 0.31.0
2020-07-08 01:50:16,690:DEBUG:certbot.main:Arguments: ['-q']
2020-07-08 01:50:16,690:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-07-08 01:50:16,722:DEBUG:certbot.log:Root logging level set at 30
2020-07-08 01:50:16,723:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-07-08 01:50:16,734:DEBUG:certbot.renewal:no renewal failures
2020-07-08 12:52:11,104:DEBUG:certbot.main:certbot version: 0.31.0
2020-07-08 12:52:11,105:DEBUG:certbot.main:Arguments: []
2020-07-08 12:52:11,105:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-07-08 12:52:11,138:DEBUG:certbot.log:Root logging level set at 20
2020-07-08 12:52:11,141:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-07-08 12:52:11,151:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2020-07-08 12:52:11,151:DEBUG:certbot.plugins.selection:No candidate plugin
2020-07-08 12:52:11,152:DEBUG:certbot.plugins.selection:Selected authenticator None and installer None
2020-07-08 12:59:24,483:DEBUG:certbot.main:certbot version: 0.31.0
2020-07-08 12:59:24,484:DEBUG:certbot.main:Arguments: []
2020-07-08 12:59:24,485:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-07-08 12:59:24,512:DEBUG:certbot.log:Root logging level set at 20
2020-07-08 12:59:24,513:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log

addon log - log was empty before manual start.

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] file-structure.sh: executing... 
[cont-init.d] file-structure.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[13:02:01] INFO: Selected DNS Provider: dns-cloudflare
[13:02:02] INFO: Use propagation seconds: 60
[13:02:02] INFO: Use CloudFlare token
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-cloudflare, Installer None
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for hassio.borpin.net
Waiting 60 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /data/letsencrypt/live/hassio.borpin.net/fullchain.pem
   Your key file has been saved at:
   /data/letsencrypt/live/hassio.borpin.net/privkey.pem
   Your cert will expire on 2020-10-06. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

syslog excerpts

Jul  8 01:50:15 hassio systemd[1]: Starting Certbot...
Jul  8 01:50:16 hassio systemd[1]: certbot.service: Succeeded.
Jul  8 01:50:16 hassio systemd[1]: Started Certbot.

/var/log/syslog:Jul  8 12:00:02 hassio CRON[15469]: (root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)

[agners]

From what I understand, the supervisor has no feature which starts the Let's Encrypt add-on when necessary... By default the startup mode is set as once, which basically means "on user request". If you set it to "Start on boot", then it will get started on boot but you will have to reboot the system periodically...

From what I can tell there is no "periodically startup mode" for add-ons (see also startup section in https://developers.home-assistant.io/docs/add-ons/configuration/). I guess a cron type of startup mode would be required for this case, which need to be implemented in the supervisor project.

[borpin]

@agners, no reason why there couldn't be AFAICS. I think if you modify the installed certbot config to find the right config file it will work, but I have not delved any deeper as yet.

I think the addon installs certbot to the underlying OS (although I may have done that in the past), but the standard config does not find the certificate config by default.

[lambtho12]

A simple (user-based) fix for this is to launch the add-on periodically to force it to renew the certificate if needed. I use the following automation to this end

# Start Let's Encrypt every night to force renewal of certificate
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt

Note that running this every day is probably overkill as certbot will renew it as soon as it is below 30 days left, but it does not really matters.

@ludeeus , maybe we can add this small snippet to the addon documentation while we wait for a more integrated solution?

[ludeeus]

That will only solve half the issue. All services (like Home Assistant) will still use the old certificate.

[agners]

@ludeeus hm, I guess because the SSL certificate is already loaded at that time? Ideally, HA Core should have a reload function to just reload the SSL certificate. I guess for add-ons there is no common way to handle that, probably just restarting them is fine.

[ludeeus]

Yeah, all weservers load the certificate on startup, and that will continue to be loaded (even if the file changes) until the server restarts.

[borpin]

Under normal circumstances, certbot restarts apache on renewal. I wonder if the addon could create a flag/sensor that other services monitor?

With an update cycle for core as it is, and therefore regular restarts, it is less of an issue, but thinking on that, I'd have expected the addon to start and update so now surprised I saw the expiry notice. Odd.

[ludeeus]

That would be something that should be handled by a certificate manager in the supervisor and not by this addon.

[borpin]

That would be something that should be handled by a certificate manager in the supervisor

Not sure what this was a reply to, but if my comment, it should, but obviously isn't.

As I said, under normal circumstances, certbot (certificate manager) does it for you.

[ludeeus]

It was in regards to this part

I wonder if the addon could create a flag/sensor that other services monitor?

[borpin]

So back to my point, under normal circumstances, the tool doing the certificate renewal, i.e. managing the certificates, would do the server restart (as certbot does if the right plugin is used). Because the certificate manager is actually an addon, it cannot directly control the restart, but the addon certainly could tell supervisor/HA it needs to restart.

Supervisor does not have a certificate manager AFAICS.

[ludeeus]

The addon is not aware of which services that uses the certificates.

Supervisor does not have a certificate manager AFAICS.

Correct, and when someone adds that to the supervisor it will know which service uses which certificate and can handle that.

For now, your best option (since only you know where you are using the certificates), an automation as described here https://github.com/home-assistant/hassio-addons/issues/1445#issuecomment-668254065 With potentially another action for restarts of the addons (and/or core) where you use the certificates.

[lambtho12]

That will only solve half the issue. All services (like Home Assistant) will still use the old certificate.

So this should do the trick instead then (using the cert_expiry integration):

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    - condition: numeric_state
      entity_id: sensor.ssl_certificate_expiry
      below: 30
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

[ludeeus]

If you only use it for homeassistant, and you can have it "randomly" restart during the night yes :+1:

[agners]

@ludeeus @borpin most web servers do also support SSL certificate reloads via signals, this avoids full blown restarts. E.g. nginx supports reload which do not lead to any downtime: https://stackoverflow.com/questions/43088363/how-nginx-reload-work-why-it-is-zero-downtime. And that is also what certbot uses by default (see https://github.com/certbot/certbot/blob/09ab4aea01aaf95a2a830ad48271aa6bd11eef84/certbot-nginx/certbot_nginx/_internal/configurator.py#L1178).

[maidau]

So this should do the trick instead then (using the cert_expiry integration):

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    - condition: numeric_state
      entity_id: sensor.ssl_certificate_expiry
      below: 30
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

@lambtho12 please excuse me, I'm new here :-) Which yaml file should that code be inserted? I currently have the simple once a day time based restart code placed in system.yaml and it doesn't seem to be doing a thing (I've checked the Let's Encrypt logs).

[lambtho12]

@maidau This is an automation. So it should be in automations.yaml or something like that. You could also recreate it using the automation editor of the UI (under configuration/automation) instead if you prefer. See the automation documentation for more information.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[asychev]

The solution above will not work anymore because of https://github.com/home-assistant/core/pull/42338

[agners]

@asychev you can use a template to calculate the time until expiry:

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    condition: template
    value_template: {{ as_timestamp(states("sensor.cert_expiry")) - as_timestamp(now()) < 30 * 24 * 60 * 60 }}
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

Update: Fixed syntax

[borpin]

So what it needs is the following;

Restart letsencrypt addon Restart NGINX addon (new cert shows in browser padlock) Restart HA to update sensor.

@agners - what is the right config to restart the nginx addon (and where could I look that up for future reference?)

[ronnix]

@agners @asychev FYI, this syntax for the condition worked better for me :

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    condition: template
    value_template: "{{as_timestamp(states('sensor.cert_expiry_timestamp_HOST_PORT')) - as_timestamp(now()) < 30 * 24 * 60 * 60}}"
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

[medmunds]

@borpin the nginx addon's name appears to be core_nginx_proxy. (I don't know the right place to look that up, but I found it by looking in the URL for that addon in the Supervisor page.)

Adding it to the automation's actions seems to restart it:

# ...
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: hassio.addon_restart
      data:
        addon: core_nginx_proxy
    - service: homeassistant.restart

(BTW, trying to call hassio.addon_restart on addon: core_nginx_proxy from the Developer Tools Services panel gives the error "Failed to call service hassio/addon_restart. undefined", but does seem to restart nginx.)

[borpin]

My solution worked overnight - thanks to all for your help

https://community.home-assistant.io/t/lets-encrypt-add-on-not-renewing-certificates-correctly/214294

[decentropy]

So what it needs is the following;

Restart letsencrypt addon Restart NGINX addon (new cert shows in browser padlock) Restart HA to update sensor.

Not sure why one needs to restart NGINX core proxy.. and then restart HA. Wouldn't restarting HA cover both?

[borpin]

Not sure why one needs to restart NGINX core proxy.. and then restart HA. Wouldn't restarting HA cover both?

Because restarting HA just restarts the core not the Supervisor (I believe). For that you need to do a full host restart (which I was trying to avoid).

Try is and see 😄

[supersebbo]

Thanks for this! I was going crazy trying to work out why CertBot hadn't updated my certificates, I assumed it was copying to the wrong place, but actually just needed to restart HA.

[mkitchingh]

Is everything here still accurate with current versions? I just experienced my first cert expiring. There was no attempt to renew it. A couple of reboots and restarting let's encrypt add-on got things working.

[borpin]

Is everything here still accurate with current versions? I just experienced my first cert expiring. There was no attempt to renew it. A couple of reboots and restarting let's encrypt add-on got things working.

As far as I know. My script runs and the certificates update. I am not aware the issue is fixed. Closed by a bot.

[mkitchingh]

As far as I know. My script runs and the certificates update. I am not aware the issue is fixed. Closed by a bot.

Thanks. Can you confirm or link to which version/script you are using? I have read through everything and there are multiple people commenting with updates, and I'm unclear what the final/best version is.

[borpin]

As far as I know. My script runs and the certificates update. I am not aware the issue is fixed. Closed by a bot.

Thanks. Can you confirm or link to which version/script you are using? I have read through everything and there are multiple people commenting with updates, and I'm unclear what the final/best version is.

Automation below. I get the days left from an obscure source! I could get it from here https://www.home-assistant.io/integrations/cert_expiry and calculate the days left with a template.

The actions are the important part.

alias: Renew SSL Certificate
description: ""
mode: single
triggers:
  - at: "03:00"
    trigger: time
conditions:
  - condition: numeric_state
    below: "30"
    entity_id: sensor.cert_expiry_xxx
actions:
  - data:
      addon: core_letsencrypt
    action: hassio.addon_restart
  - delay:
      hours: 0
      minutes: 5
      seconds: 0
      milliseconds: 0
  - data:
      addon: core_nginx_proxy
    action: hassio.addon_restart
  - delay:
      hours: 0
      minutes: 1
      seconds: 0
      milliseconds: 0
  - data: {}
    action: homeassistant.restart