eclipse / mosquitto

Eclipse Mosquitto - An open source MQTT broker
https://mosquitto.org
Other
8.99k stars 2.39k forks source link

Installing mosquitto fails because service fails to start #1419

Closed raman325 closed 4 years ago

raman325 commented 5 years ago

When attempting to install mosquitto, the installation fails because the service fails to start due to a segmentation fault. I've tried searching other issues but the issues I were able to find were resolved, and I couldn't find anything obvious to do from the threads, which is why I opened a new issue. Can someone help me figure out how to resolve this?

I've posted all the relevant details I could think of below but let me know if there's anything I can provide.

OS: Raspbian GNU/Linux 10 (buster) mosquitto version: Mosquitto MQTT v3.1/v3.1.1 Broker mosquitto package: 1.6.6-0mosquitto1~buster1

Installation logs:

$ apt-cache search mosquitto; sudo apt-get update; sudo apt-get install -f libmosquitto-dev mosquitto mosquitto-clients libmosquitto1
libmosquittopp-dev - MQTT version 3.1 client C++ library, development files
libmosquittopp1 - MQTT version 3.1/3.1.1 client C++ library
mosquitto-dev - Development files for Mosquitto
libmosquitto-dev - MQTT version 3.1/3.1.1/5.0 client library, development files
libmosquitto1 - MQTT version 3.1/3.1.1/5.0 client library
libmosquitto1-dbgsym - debug symbols for libmosquitto1
libmosquittopp1-dbgsym - debug symbols for libmosquittopp1
mosquitto - MQTT version 3.1/3.1.1/5.0 compatible message broker
mosquitto-clients - Mosquitto command line MQTT clients
mosquitto-clients-dbgsym - debug symbols for mosquitto-clients
mosquitto-dbgsym - debug symbols for mosquitto
Hit:1 http://raspbian.raspberrypi.org/raspbian buster InRelease
Hit:2 http://archive.raspberrypi.org/debian buster InRelease
Hit:3 https://repo.mosquitto.org/debian buster InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libev4 libuv1 libwebsockets8
Suggested packages:
  apparmor
The following NEW packages will be installed:
  libev4 libmosquitto-dev libmosquitto1 libuv1 libwebsockets8 mosquitto mosquitto-clients
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 610 kB of archives.
After this operation, 1327 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirror.umd.edu/raspbian/raspbian buster/main armhf libev4 armhf 1:4.25-1 [34.5 kB]
Get:2 http://mirror.umd.edu/raspbian/raspbian buster/main armhf libuv1 armhf 1.24.1-1 [96.7 kB]
Get:3 http://mirror.umd.edu/raspbian/raspbian buster/main armhf libwebsockets8 armhf 2.0.3-3 [85.6 kB]
Get:4 https://repo.mosquitto.org/debian buster/main armhf libmosquitto1 armhf 1.6.6-0mosquitto1~buster1 [70.9 kB]
Get:5 https://repo.mosquitto.org/debian buster/main armhf libmosquitto-dev armhf 1.6.6-0mosquitto1~buster1 [62.1 kB]
Get:6 https://repo.mosquitto.org/debian buster/main armhf mosquitto armhf 1.6.6-0mosquitto1~buster1 [172 kB]
Get:7 https://repo.mosquitto.org/debian buster/main armhf mosquitto-clients armhf 1.6.6-0mosquitto1~buster1 [88.2 kB]
Fetched 610 kB in 6s (96.2 kB/s)
Selecting previously unselected package libev4:armhf.
(Reading database ... 46206 files and directories currently installed.)
Preparing to unpack .../0-libev4_1%3a4.25-1_armhf.deb ...
Unpacking libev4:armhf (1:4.25-1) ...
Selecting previously unselected package libmosquitto1:armhf.
Preparing to unpack .../1-libmosquitto1_1.6.6-0mosquitto1~buster1_armhf.deb ...
Unpacking libmosquitto1:armhf (1.6.6-0mosquitto1~buster1) ...
Selecting previously unselected package libmosquitto-dev:armhf.
Preparing to unpack .../2-libmosquitto-dev_1.6.6-0mosquitto1~buster1_armhf.deb ...
Unpacking libmosquitto-dev:armhf (1.6.6-0mosquitto1~buster1) ...
Selecting previously unselected package libuv1:armhf.
Preparing to unpack .../3-libuv1_1.24.1-1_armhf.deb ...
Unpacking libuv1:armhf (1.24.1-1) ...
Selecting previously unselected package libwebsockets8:armhf.
Preparing to unpack .../4-libwebsockets8_2.0.3-3_armhf.deb ...
Unpacking libwebsockets8:armhf (2.0.3-3) ...
Selecting previously unselected package mosquitto.
Preparing to unpack .../5-mosquitto_1.6.6-0mosquitto1~buster1_armhf.deb ...
Unpacking mosquitto (1.6.6-0mosquitto1~buster1) ...
Selecting previously unselected package mosquitto-clients.
Preparing to unpack .../6-mosquitto-clients_1.6.6-0mosquitto1~buster1_armhf.deb ...
Unpacking mosquitto-clients (1.6.6-0mosquitto1~buster1) ...
Setting up libmosquitto1:armhf (1.6.6-0mosquitto1~buster1) ...
Setting up libev4:armhf (1:4.25-1) ...
Setting up mosquitto-clients (1.6.6-0mosquitto1~buster1) ...
Setting up libuv1:armhf (1.24.1-1) ...
Setting up libmosquitto-dev:armhf (1.6.6-0mosquitto1~buster1) ...
Setting up libwebsockets8:armhf (2.0.3-3) ...
Setting up mosquitto (1.6.6-0mosquitto1~buster1) ...
Job for mosquitto.service failed because a fatal signal was delivered to the control process.
See "systemctl status mosquitto.service" and "journalctl -xe" for details.
invoke-rc.d: initscript mosquitto, action "restart" failed.
● mosquitto.service - Mosquitto MQTT v3.1/v3.1.1 Broker
   Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: signal) since Sun 2019-09-22 02:57:45 EDT; 200ms ago
     Docs: man:mosquitto.conf(5)
           man:mosquitto(8)
  Process: 28178 ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf (code=killed, signal=SEGV)
 Main PID: 28178 (code=killed, signal=SEGV)

Sep 22 02:57:45 pihole systemd[1]: mosquitto.service: Service RestartSec=100ms expired, scheduling restart.
Sep 22 02:57:45 pihole systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 1.
Sep 22 02:57:45 pihole systemd[1]: Stopped Mosquitto MQTT v3.1/v3.1.1 Broker.
dpkg: error processing package mosquitto (--configure):
 installed mosquitto package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (241-7~deb10u1+rpi1) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10+rpi1) ...
Errors were encountered while processing:
 mosquitto
E: Sub-process /usr/bin/dpkg returned an error code (1)

Running mosquitto manually:

$ /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
Segmentation fault

/etc/mosquitto/mosquitto.conf contents:

# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example

pid_file /var/run/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d
ralight commented 5 years ago

I've just tested it on my pi 3, and it installed fine. Does it crash just the same if you just run it as mosquitto? Could you also try install mosquitto-dbgsym, then running:

gdb --args /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
(gdb) run
(gdb) # it should crash
(gdb) bt

Then please post the back trace from the bt command.

raman325 commented 5 years ago

I should mention that I am running this on a Pi Zero W.

pi@pihole:~ $ gdb --args /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
(gdb) run
Starting program: /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x004533da in ?? ()
(gdb) bt
#0  0x004533da in ?? ()
#1  0x00404226 in _start () at mosquitto.c:390
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
raman325 commented 5 years ago

Not sure if this helps but #696 was the issue I started reading first before posting

vandecook commented 5 years ago

The same problem here.

OS: Raspbian GNU/Linux 10 (buster) mosquitto package: 1.6.6-0mosquitto1~buster1 Raspberry 1 B

Installation stops with (sorry german)

mosquitto (1.6.6-0mosquitto1~buster1) wird eingerichtet ...
Job for mosquitto.service failed because a fatal signal was delivered to the control process.
See "systemctl status mosquitto.service" and "journalctl -xe" for details.
invoke-rc.d: initscript mosquitto, action "start" failed.
● mosquitto.service - Mosquitto MQTT v3.1/v3.1.1 Broker
   Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: signal) since Mon 2019-09-23 08:34:31 CEST; 155ms ago
     Docs: man:mosquitto.conf(5)
           man:mosquitto(8)
  Process: 1465 ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf (code=killed, signal=SEGV)
 Main PID: 1465 (code=killed, signal=SEGV)

Sep 23 08:34:31 testRaspiV1 systemd[1]: mosquitto.service: Service RestartSec=100ms expired, scheduling restart.
Sep 23 08:34:31 testRaspiV1 systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 1.
Sep 23 08:34:32 testRaspiV1 systemd[1]: Stopped Mosquitto MQTT v3.1/v3.1.1 Broker.

Debugging

gdb --args /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
GNU gdb (Raspbian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/mosquitto...Reading symbols from /usr/lib/debug/.build-id/e4/d60a41e8d2c0473ebfe145b28b2c3d103d0cb7.debug...done.
done.
(gdb) run
Starting program: /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x004533da in ?? ()
(gdb) 

Tried to reinstall several times.

klavertje44 commented 5 years ago

Same issue here. Try to remove and install several times. Help!

Raspberry 2B Raspbian Stretch 9.11

After: sudo apt-get update sudo apt-get upgrade

pi@raspberrypi:~ $ sudo apt-get upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 1 not fully installed or removed. After this operation, 0 B of additional disk space will be used. Do you want to continue? [Y/n] Y Setting up mosquitto (1.6.6-0mosquitto1~stretch1) ... Job for mosquitto.service failed because a fatal signal was delivered to the control process. See "systemctl status mosquitto.service" and "journalctl -xe" for details. invoke-rc.d: initscript mosquitto, action "restart" failed. ● mosquitto.service - Mosquitto MQTT v3.1/v3.1.1 Broker Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; vendor preset: enabled) Active: activating (start) since Tue 2019-09-24 19:46:39 CEST; 359ms ago Docs: man:mosquitto.conf(5) man:mosquitto(8) Main PID: 5167 CGroup: /system.slice/mosquitto.service

Sep 24 19:46:39 raspberrypi systemd[1]: Starting Mosquitto MQTT v3.1/v3.1.1 Broker... Sep 24 19:46:39 raspberrypi systemd[1]: mosquitto.service: Main process exited, code=killed, status=11/SEGV Sep 24 19:46:39 raspberrypi systemd[1]: Failed to start Mosquitto MQTT v3.1/v3.1.1 Broker. Sep 24 19:46:39 raspberrypi systemd[1]: mosquitto.service: Unit entered failed state. Sep 24 19:46:39 raspberrypi systemd[1]: mosquitto.service: Failed with result 'signal'. dpkg: error processing package mosquitto (--configure): subprocess installed post-installation script returned error exit status 1 Errors were encountered while processing: mosquitto E: Sub-process /usr/bin/dpkg returned an error code (1)

pi@raspberrypi:~ $ gdb --args /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-linux-gnueabihf". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/mosquitto...Reading symbols from /usr/lib/debug/.build-id/d9/0129e31e29521246f7675c58f527377b8795dd.debug...done. done. (gdb) run Starting program: /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault. 0x00453c6e in ?? () (gdb) bt

0 0x00453c6e in ?? ()

1 0x00404aba in _start ()

Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb)

hherm commented 5 years ago

I ran into the same issue. Just updated a working mosquitto to version: 1.6.7-0mosquitto1~stretch1 System: Raspberry Pi 2B with Raspian Stretch Repro: deb https://repo.mosquitto.org/debian stretch main

Systemctl-output:

Sep 25 19:59:03 lager systemd[1]: Starting Mosquitto MQTT v3.1/v3.1.1 Broker...
Sep 25 19:59:03 lager systemd[1]: mosquitto.service: Main process exited, code=killed, status=11/SEGV
Sep 25 19:59:03 lager systemd[1]: Failed to start Mosquitto MQTT v3.1/v3.1.1 Broker.
Sep 25 19:59:03 lager systemd[1]: mosquitto.service: Unit entered failed state.
Sep 25 19:59:03 lager systemd[1]: mosquitto.service: Failed with result 'signal'.
dpkg: Fehler beim Bearbeiten des Paketes mosquitto (--configure):
Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
Fehler traten auf beim Bearbeiten von:
mosquitto

Running mosquitto manually gives: "Speicherzugriffsfehler" (Segmentation fault in EN)

Trying to reinstall is not better:

$ sudo apt-get --reinstall install mosquitto
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
0 aktualisiert, 0 neu installiert, 1 erneut installiert, 0 zu entfernen und 0 nicht aktualisiert.
1 nicht vollständig installiert oder entfernt.
Nach dieser Operation werden 0 B Plattenplatz zusätzlich benutzt.
E: Internal Error, No file name for mosquitto:armhf

To purge mosquitto and install it again does not change anything. Same error from top.

Any suggestions?

Iedjee commented 5 years ago

I have the same problem coming from 1.6.4 on a Raspberry Pi 1B. I managed to get back to a working Mosquitto by sudo apt-get install mosquitto=1.6.4-0mosquitto1~stretch1

Now i set the package on hold with sudo apt-mark hold mosquitto

I hope it gets resolved.

klavertje44 commented 5 years ago

Yes, I didn't know how to downgrade, but this is working. Thank you verry much

Now waiting for the fix in the next release.

raman325 commented 5 years ago

Thanks @Iedjee this version is working for me. I am on buster so I ran sudo apt-get install mosquitto=1.6.4-0mosquitto1~buster1

hherm commented 5 years ago

I managed to get version 1.6.7 working by installing some libraries, cloning and compiling myself. After that, I exchanged the mosquitto executeable with the fresh compiled one. Works like a charm. :-)

Installed: $ sudo apt install libssl-dev libgnutls28-dev libsystemd-dev xsltproc

PS: The latter one is for documentation only, so not strictly necessary, cause the object of interest is: src/mosquitto.

simonk83 commented 5 years ago

Same issue here on a Pi Zero W. Rolling back to an earlier version hasn't helped...

rafuz commented 5 years ago

Same error on Pi Zero W. Rolled back to 1.6.4 stretch and it is working again.

simonk83 commented 5 years ago

Rolling back doesn't seem to be working for me, anything else I can try?

hherm commented 5 years ago

Compiling on your Pi should work, too. But maybe the problem is located elsewhere, if version 1.6.4 doesn't work for you.

simonk83 commented 5 years ago

All good, I didn't realise I had to roll back libmosquitto etc as well, it's back working now

pjmsullivan commented 5 years ago

For anyone whose Raspberry Pi Zero with Monitor stopped working after an update, downgrade as @ledjee recommended:

sudo apt-get install mosquitto=1.6.4-0mosquitto1~stretch1

Additionally, I downgraded the libmosquitto1 and mosquitto-clients packages in that order as follows: sudo apt-get install libmosquitto1=1.6.4-0mosquitto1~stretch1 sudo apt-get install mosquitto-clients=1.6.4-0mosquitto1~stretch1

Thanks for the assistance @simonk83 and @ledjee

aric89 commented 5 years ago

@pjmsullivan I updated my zeros tonight and it broke. I ran the 3 commands that you posted and i can verify that fixes it. Thanks!

Nephiel commented 4 years ago

Same here, 1.6.7 on Pi Zero W is not working. Same with 1.6.6. Looks like these builds are just not compatible with older Pi armhf. I'm downgrading to 1.6.4 for now.

26tajeen commented 4 years ago

I'm trying all this on a Ra Pi Zero W after doing a "sudo apt upgrade forked-daapd" it then decided to upgrade to1.6.7-0mosquitto1~stretch1which then resulted in Job for mosquitto.service failed because a fatal signal was delivered to the control process. I've tried all the commands @pjmsullivan mentioned but I still get systemd[1]: mosquitto.service: Failed with result 'signal'.

Nephiel commented 4 years ago

I upgraded to 1.6.8 on a Pi Zero W, and the issue seems to be fixed.

ralight commented 4 years ago

Good. The build environment for the raspberry pi has been recreated to fix this.

ahmed305salah commented 4 years ago

i have a similar problem on starting mosquitto this is what i typed

sudo /etc/init.d/mosquitto start

this shows [....] Starting mosquitto (via systemctl): mosquitto.serviceJob for mosquitto.service failed because the control process exited with error code. See "systemctl status mosquitto.service" and "journalctl -xe" for details. failed! i am using RPi 4 B

gdemarchi commented 4 years ago

For the records, since it took quite some time to me to find the solution.

My service mosquitto was dying too soon as well,

'raspberrypi systemd[1]: mosquitto.service: Service RestartSec=100ms expired, scheduling restart. raspberrypi systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5. raspberrypi systemd[1]: Stopped Mosquitto MQTT v3.1/v3.1.1 Broker. raspberrypi systemd[1]: mosquitto.service: Start request repeated too quickly. raspberrypi systemd[1]: mosquitto.service: Failed with result 'exit-code'. raspberrypi systemd[1]: Failed to start Mosquitto MQTT v3.1/v3.1.1 Broker.'

and the simple reason for this "too quickly" was that the port used by mosquitto was already busy:

sudo less /var/log/mosquitto/mosquitto.log

shows:

1586870339: Config loaded from /etc/mosquitto/mosquitto.conf. 1586870339: Opening ipv4 listen socket on port 1883. 1586870339: Error: Address already in use 1586870513: mosquitto version 1.6.8 starting

The reason is that I have a docker instance with an hassio image running, and in the home assistant configuration I had the mosquitto plugin enabled. I've disabled it from the Lovelace/Supervisor page, and this action freed up the port 1883. After this I reinstalled mosquitto (probably restarting it would be fine enough), and it worked like a charm ...

WayeeWang commented 3 years ago

Thank you @gdemarchi ! Exactly the same cause for me. After disabling it in HA, all working fine now

createcandle commented 2 years ago

I kept running into this issue. For me this was the solution:

log_dest stdout

per_listener_settings true

listener 1883
allow_anonymous true
persistence false
max_connections 20
max_queued_messages 100
max_inflight_messages 50
password_file /home/pi/mounted_drive/etc/mosquitto/mosquitto_users

I really wish there was some more useful documentation and some more example files.