Broker failing to start after running for a few weeks

[jimmy232]

Mosquitto is installed through apt-get and has been working well for ages. Now I've I've added a few extra programs such as Sambo, raspAP, SIXFAB and maybe a few more programs and now mosquito broker is failing to start on random pi's.

Some help would be much appreciated to resolve this issue.

~ $ systemctl status mosquitto.service ● mosquitto.service - Mosquitto MQTT v3.1/v3.1.1 Broker Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2020-12-18 20:21:25 AEST; 9s ago Docs: man:mosquitto.conf(5) man:mosquitto(8) Process: 1768 ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf (code=exited, status=1/FAILURE) Main PID: 1768 (code=exited, status=1/FAILURE)

Dec 18 20:21:25 MoniconPLC systemd[1]: mosquitto.service: Service RestartSec=100ms expired, scheduling restart. Dec 18 20:21:25 MoniconPLC systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5. Dec 18 20:21:25 MoniconPLC systemd[1]: Stopped Mosquitto MQTT v3.1/v3.1.1 Broker. Dec 18 20:21:25 MoniconPLC systemd[1]: mosquitto.service: Start request repeated too quickly. Dec 18 20:21:25 MoniconPLC systemd[1]: mosquitto.service: Failed with result 'exit-code'. Dec 18 20:21:25 MoniconPLC systemd[1]: Failed to start Mosquitto MQTT v3.1/v3.1.1 Broker.

~ $ sudo systemctl start mosquitto.service Job for mosquitto.service failed because the control process exited with error code. See "systemctl status mosquitto.service" and "journalctl -xe" for details.

$ sudo /usr/sbin/mosquitto 1608287403: mosquitto version 1.5.7 starting 1608287403: Using default config. 1608287404: Opening ipv4 listen socket on port 1883. 1608287404: Opening ipv6 listen socket on port 1883. 1608287415: New connection from 10.0.0.28 on port 1883. 1608287415: New client connected from 10.0.0.28 as mqtt_f6072646.c65a78 (c1, k60). 1608287415: New connection from 10.0.0.28 on port 1883. 1608287415: New client connected from 10.0.0.28 as mqtt_dc198b3b.082108 (c1, k60). ^C1608287448: mosquitto version 1.5.7 terminating

[ralight]

Does running the below command give any more hints? That is the way that systemd will run it.

sudo /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

[jimmy232]

I tried this command after reading through existing issues... turns out the command is invalid.

~ $ /usr/sbin/mosquitto /etc/mosquitto/mosquitto.conf Error: Unknown option '/etc/mosquitto/mosquitto.conf'. mosquitto version 1.5.7

mosquitto is an MQTT v3.1.1 broker.

Usage: mosquitto [-c config_file] [-d] [-h] [-p port]

-c : specify the broker config file. -d : put the broker into the background after starting. -h : display this help. -p : start the broker listening on the specified port. Not recommended in conjunction with the -c option. -v : verbose mode - enable all logging types. This overrides any logging options given in the config file.

See http://mosquitto.org/ for more information.

On Fri, Dec 18, 2020 at 9:04 PM Roger Light notifications@github.com wrote:

Does running the below command give any more hints? That is the way that systemd will run it.

sudo /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/mosquitto/issues/1965#issuecomment-748025026, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIYB6I5IYE2ZUJPYDDJ7HDSVMZL5ANCNFSM4VA72HGA .

[jimmy232]

Sorry, looks like I was missing the parameter -c

Here is the error message I receive. I do know uninstalling and reinstalling fixes the issue, but this option is no good for remote servers...

~ $ sudo /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf 1608291244: Error: Unable to open log file /var/log/mosquitto/mosquitto.log for writing

On Fri, Dec 18, 2020 at 9:33 PM James Currie jamescurriehv@gmail.com wrote:

I tried this command after reading through existing issues... turns out the command is invalid.

~ $ /usr/sbin/mosquitto /etc/mosquitto/mosquitto.conf Error: Unknown option '/etc/mosquitto/mosquitto.conf'. mosquitto version 1.5.7

mosquitto is an MQTT v3.1.1 broker.

Usage: mosquitto [-c config_file] [-d] [-h] [-p port]

-c : specify the broker config file. -d : put the broker into the background after starting. -h : display this help. -p : start the broker listening on the specified port. Not recommended in conjunction with the -c option. -v : verbose mode - enable all logging types. This overrides any logging options given in the config file.

See http://mosquitto.org/ for more information.

On Fri, Dec 18, 2020 at 9:04 PM Roger Light notifications@github.com wrote:

Does running the below command give any more hints? That is the way that systemd will run it.

sudo /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/mosquitto/issues/1965#issuecomment-748025026, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIYB6I5IYE2ZUJPYDDJ7HDSVMZL5ANCNFSM4VA72HGA .

[ralight]

Ok, so it seems like either the /var/log/mosquitto directory, or the file /var/log/mosquitto/mosquitto.log isn't owned by the mosquitto user, or doesn't have write permission.

One of more of these commands will fix that for you:

sudo chown mosquitto /var/log/mosquitto
sudo chown mosquitto /var/logl/mosquitto/mosquitto.log
sudo chmod u+w /var/log/mosquitto/
sudo chmod u+w /var/log/mosquitto/mosquitto.log

[jimmy232]

Ok, it turns out that second command sudo chown mosquitto /var/log/mosquitto/mosquitto.log grants access and then the following command sudo /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf brings Mosquitto back to life. There are no messages echoed back and when rebooted Mosquitto Broker fails to connect again requiring the same two commands to get the broker to work.

sudo chown mosquitto /var/log/mosquitto/mosquitto.log

sudo /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

[image: image.png]

On Fri, Dec 18, 2020 at 10:35 PM Roger Light notifications@github.com wrote:

Ok, so it seems like either the /var/log/mosquitto directory, or the file /var/log/mosquitto/mosquitto.log isn't owned by the mosquitto user, or doesn't have write permission.

One of more of these commands will fix that for you:

sudo chown mosquitto /var/log/mosquitto sudo chown mosquitto /var/logl/mosquitto/mosquitto.log sudo chmod u+w /var/log/mosquitto/ sudo chmod u+w /var/log/mosquitto/mosquitto.log

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/mosquitto/issues/1965#issuecomment-748061786, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIYB6NQ5MBBSJVLZU4TYKDSVNEAPANCNFSM4VA72HGA .

[Cast39]

I had a simular problem, but in my case the /var/log/mosquitto/mosquitto.log file was just missing. After manually adding it it worked fine for some time the problem came back multiple times. My best guess is that the logrotate service is not creating a new file because the configuration file in /etc/logrotate.d/mosquitto has "nocreate" in it which will prevent the creation of a new logfile. I think changing nocreate to create could fix. but I'm not sure if this is a workaround or a bug in mosquitto

[ralight]

If the broker cannot create files in /var/log/mosquitto, then the problem is most likely down to permissions on that folder itself, not the files in it. Is it owned by mosquitto?

[jimmy232]

I have checked user privileges on two different systems with the exact same problem. System 1 has the exact same privileges as system 2.

After entering system 1 with the following commands and initiated x2 reboots, everything came good again. sudo chown mosquitto /var/log/mosquitto/mosquitto.log

sudo /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

System 1 after entering the above commands [image: image.png]

System 2 [image: image.png]

There has to be something bigger at play here......

On Tue, Dec 22, 2020 at 8:40 AM Roger Light notifications@github.com wrote:

If the broker cannot create files in /var/log/mosquitto, then the problem is most likely down to permissions on that folder itself, not the files in it. Is it owned by mosquitto?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/mosquitto/issues/1965#issuecomment-749237476, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIYB6LBOHCWWJGM4U4KU3TSV7FERANCNFSM4VA72HGA .

[ralight]

Your images haven't been added.