bitraf / infrastructure

Infrastructure stuff for Bitraf's sysadmins
1 stars 3 forks source link

mqtt: service didn't work after power outage #180

Open tingox opened 1 month ago

tingox commented 1 month ago

after a power outage today, the mqtt service didn't work when the server (bomba) came up again. The relevant services mqtt and traefik (reverse proxy) run in containers (docker) on the server, both containers had started ok, and everything looked normal there.

The problem was manually worked around by restarting the containers in this order

sudo docker restart mosquitto_mosquitto_1
sudo docker restart traefik_traefik_1

(some time (a few minutes) passed between these two actions.)

Speculation / theory: the order in which the containers are started matters. If traefik is started before mqtt, the service will not work.

Anyway, the root cause should be found, and the problem fixed permanently.