let's encrypt, unable to load server certificate cert.pem

[SunSDSE]

I am using Let's Encrypt docker container working with Nginx Webserver. The let's encrypt created the SSL certs and they work fine in the Nginx Webserver. They are hosted on a NAS shared mount point.

These cert's are created and stored in a shared NFS service and bound a volume: /etc/letencrypt/live/{URI}/
All files cert.pem, chain.pem, full chain.pem and privkey.pem are written by root and owned by root.

When I configure the mosquitto-eclipse to use the same certificates I get an unable to load the server certificate.

When mosquitto-eclipse docker container starts it is unable to read the privkey as the protections are rw for root only. If I change the owner to be mosquitto:root example -rw------- 1 1883 root 1704 Mar 15 12:43 privkey.pem Then the mosquitto MQTT docker container will start and pass messages.

The next time the Certbot will renew the certificates this will break the MQTT server as the permissions will be set back to root:root ownership.

Is there any documents that discuss the proper integration of let's encrypt and this ownership issue?

[ralight]

My normal suggestion would be to copy the certificates to a location that mosquitto can read and change the ownership as appropriate, using a letsencrypt post update hook. If you're mounting on NFS for your own reason that may not be suitable, but you could still use a hook to set the permissions.

You can see the example script which does the copying here, it should be reasonably easy to modify to your own needs: https://github.com/eclipse/mosquitto/blob/master/misc/letsencrypt/mosquitto-copy.sh

[mattst88]

I think this worked before v2.0.0 because mosquitto dropped privileges later:

See: https://github.com/eclipse/mosquitto/blob/master/ChangeLog.txt#L336

- If Mosquitto is run on as root on a unix like system, it will attempt to
  drop privileges as soon as the configuration file has been read. This is in
  contrast to the previous behaviour where elevated privileges were only
  dropped after listeners had been started (and hence TLS certificates loaded)
  and logging had been started. The change means that clients will never be
  able to connect to the broker when it is running as root, unless the user
  explicitly sets it to run as root, which is not advised. It also means that
  all locations that the broker needs to access must be available to the
  unprivileged user. In particular those people using TLS certificates from
  Lets Encrypt will need to do something to allow Mosquitto to access
  those certificates. An example deploy renewal hook script to help with this
  is at `misc/letsencrypt/mosquitto-copy.sh`.

I assume it's not possible to separate starting the listeners from reading the TLS certs, so there's no opportunity to drop privileges in between?

[ralight]

@mattst88 The problem with that approach is that after you've dropped privileges you can't load anything that is root only. What that means is that when your 90 day Lets Encrypt certificate with root only access expires, you have to completely restart the broker. With the arrangement as of now, you have to make sure your certificates are accessible from the outset, or put in place a process to ensure this happens after renewal. After you've done that, it's plain sailing - the broker can remain running and you can send a SIGHUP to tell it to reload the certificates.

[mattst88]

Ah, that makes sense. Thanks!

[ralight]

@SunSDSE Are you happy for this to be closed now?

[SunSDSE]

You raise a good question.

I came to this issue because I am using the eclipse-mosquitto container using Let’s Encrypt Certs. The environment with certbot had been working, but stopped when the change was made. I will work with my environment to integrate these changes into my docker/docker-compose files, but it really should be included in the documentation found at https://hub.docker.com/_/eclipse-mosquitto https://hub.docker.com/_/eclipse-mosquitto

As this will affect many more people.

My Docker Swarm is using NGINX, Certbot, and eclipse-mosquitto. This change will affect how all three docker container work together. https://hub.docker.com/r/certbot/certbot https://hub.docker.com/r/certbot/certbot https://certbot.eff.org/docs/install.html#running-with-docker https://certbot.eff.org/docs/install.html#running-with-docker

One challenge that I have yet to determine. Sending a signal HUP to a mosquitto daemon is fine when running on a single node. But in my swarm environment I need to issue a docker service restart on the mosquitto as I will not know what docker worker node the mosquitto container is running on.

I would expect that a script running inside a docker container could not restart (or control) a docker service that is managing the docker swarm that is running the container issuing the script. There are some potential security issues with this. As most nodes in a docker swarm are configured as workers and not managers.

Thank you all for the assistance in this matter.

John

On Apr 15, 2021, at 5:17 PM, Roger Light @.***> wrote:

@SunSDSE https://github.com/SunSDSE Are you happy for this to be closed now?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eclipse/mosquitto/issues/2161#issuecomment-820819768, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHG6MLHIGZFLCCIIUCLSAODTI562XANCNFSM42GAXEYA.

[ralight]

@SunSDSE prior to version 2 the only way to deal with the certificate expiring was to restart the broker, so if you can't send a sighup then nothing has changed.

[barndawgie]

I'm able to run a number of other containers (e.g. Home Assistant, Portainer) without jumping through these hoops - just bind the directory for my certificates to the /certs path in the container and away I go. Would be great to see a change here to allow what seems like a pretty common access pattern.

[jishi]

Ran into the same issue, but adding

user root

to the mosquitto.conf removes the privilege drop in the container, which might be "good enough" for a homelab. That should allow a SIGHUP to reload newly minted certificates as well, without restarting the container.

Be vary of security implications etc here ⚠️

[tukusejssirs]

I want to use the same certificate files (at least the CA) in both Nginx and Mosquitto Docker containers, however, when using them in Mosquitto, Mosquitto changes their ownership to 1883:1883, which makes them inaccessible to Nginx.

I don’t want to create a copy of the certificates. I want to generate them in a single place, then bind-mount them to the Docker containers. Is it possible ATM? :thinking:

---

Update

A workaround is to modify /docker-entrypoint.sh to copy the certificate files (I my case bind-mounted to /cert) to /mosquitto.

#!/bin/ash
set -e

# Set permissions
user="$(id -u)"

if [ "$user" = '0' ]; then
  [ -d "/mosquitto" ] && \
    cp -r /cert /mosquitto/cert && \
    chown -R mosquitto:mosquitto /mosquitto || true
fi

exec "$@"

Save this on your host, make it exectable (chmod a+x $filename) and update the Docker Compose file to bind-mount the certificate folder (to /cert in this example) and the /docker-entrypoint.sh.

After this change, the ownership nor permissions of the original files (on host) are not changed and Mosquitto will be happy that it can change them under /mosquitto.

Do you see any security concerns? :thinking: