search

home-assistant / addons

:heavy_plus_sign: Docker add-ons for Home Assistant
https://home-assistant.io/hassio/
Apache License 2.0
1.48k stars 1.45k forks source link

MQTT won't start after HA OS upgrade #3264

Closed blacknell closed 4 months ago

blacknell commented 9 months ago

Describe the issue you are experiencing

After upgrading to HA OS 11.0 MQTT addon won't start

Relates issue Relates #3253

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Which add-on are you reporting an issue with?

Mosquitto broker

What is the version of the add-on?

6.3.1

Steps to reproduce the issue

  1. Upgraded to 11.0
  2. Restart fails

  3. ...

System Health information

System Information

version core-2023.10.0
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.11.5
os_name Linux
os_version 6.1.21-v8
arch aarch64
timezone Europe/London
config_dir /config
Home Assistant Community Store GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 4998 Installed Version | 1.32.1 Stage | running Available Repositories | 1312 Downloaded Repositories | 8
Home Assistant Cloud logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Home Assistant Supervisor host_os | Home Assistant OS 11.0 -- | -- update_channel | stable supervisor_version | supervisor-2023.10.0 agent_version | 1.6.0 docker_version | 24.0.6 disk_total | 458.4 GB disk_used | 77.1 GB healthy | true supported | true board | rpi4-64 supervisor_api | ok version_api | ok installed_addons | Samba share (10.0.2), Mosquitto broker (6.3.1), Duck DNS (1.15.0), Check Home Assistant configuration (3.11.0), Terminal & SSH (9.7.1), Home Assistant Google Drive Backup (0.111.1), Zigbee2MQTT (1.33.1-1), InfluxDB (4.8.0), File editor (5.6.0), Grafana (9.1.0), ESPHome (2023.9.3)
Dashboards dashboards | 15 -- | -- resources | 6 views | 36 mode | storage
Recorder oldest_recorder_run | 5 October 2023 at 05:49 -- | -- current_recorder_run | 17 October 2023 at 10:11 estimated_db_size | 1369.58 MiB database_engine | sqlite database_version | 3.41.2

Anything in the Supervisor logs that might be useful for us?

2023-10-17 12:09:02: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:03] INFO: Service restart after closing
[12:09:04] INFO: Starting mosquitto MQTT broker...
1697540944: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:04: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:04] INFO: Service restart after closing
[12:09:06] INFO: Starting mosquitto MQTT broker...
1697540946: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:06: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:06] INFO: Service restart after closing
[12:09:07] INFO: Starting mosquitto MQTT broker...
1697540947: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:07: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:08] INFO: Service restart after closing
[12:09:09] INFO: Starting mosquitto MQTT broker...
1697540949: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:09: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:09] INFO: Service restart after closing
[12:09:11] INFO: Starting mosquitto MQTT broker...
1697540951: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:11: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:11] INFO: Service restart after closing
[12:09:12] INFO: Successfully send discovery information to Home Assistant.
[12:09:12] INFO: Starting mosquitto MQTT broker...
1697540952: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:12: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[12:09:13] INFO: Successfully send service information to the Supervisor.
[11:09:13] INFO: Service restart after closing
[12:09:14] INFO: Successfully send discovery information to Home Assistant.
[12:09:14] INFO: Starting mosquitto MQTT broker...
1697540954: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:14: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:15] INFO: Service restart after closing
[12:09:16] INFO: Successfully send service information to the Supervisor.
[12:09:16] INFO: Successfully send discovery information to Home Assistant.
[12:09:17] INFO: Starting mosquitto MQTT broker...
1697540957: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:17: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:18] INFO: Service restart after closing
[12:09:18] INFO: Successfully send service information to the Supervisor.
[12:09:18] INFO: Successfully send discovery information to Home Assistant.
[12:09:19] INFO: Starting mosquitto MQTT broker...
1697540959: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:19: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:20] INFO: Service restart after closing
[12:09:21] INFO: Successfully send service information to the Supervisor.
[12:09:21] INFO: Successfully send discovery information to Home Assistant.
[12:09:21] INFO: Starting mosquitto MQTT broker...
1697540961: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:21: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:22] INFO: Service restart after closing
[12:09:23] INFO: Successfully send discovery information to Home Assistant.
[12:09:23] INFO: Successfully send service information to the Supervisor.
[12:09:24] INFO: Starting mosquitto MQTT broker...
1697540964: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:24: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[12:09:24] INFO: Successfully send discovery information to Home Assistant.
[11:09:25] INFO: Service restart after closing
[12:09:26] INFO: Successfully send service information to the Supervisor.
[12:09:26] INFO: Starting mosquitto MQTT broker...
1697540966: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:26: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:27] INFO: Service restart after closing
[12:09:28] INFO: Successfully send discovery information to Home Assistant.
[12:09:28] INFO: Successfully send service information to the Supervisor.
[12:09:29] INFO: Starting mosquitto MQTT broker...
1697540969: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:29: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[12:09:29] INFO: Successfully send discovery information to Home Assistant.
[11:09:30] INFO: Service restart after closing
[12:09:31] INFO: Successfully send service information to the Supervisor.
[12:09:31] INFO: Starting mosquitto MQTT broker...
1697540971: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:31: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[12:09:32] INFO: Successfully send discovery information to Home Assistant.
[11:09:33] INFO: Service restart after closing
[12:09:33] INFO: Successfully send discovery information to Home Assistant.
[12:09:33] INFO: Successfully send service information to the Supervisor.
[12:09:34] INFO: Starting mosquitto MQTT broker...
1697540974: Loading config file /share/mosquitto/mqtt.conf
2023-10-17 12:09:34: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[11:09:35] INFO: Service restart after closing
[12:09:36] INFO: Successfully send discovery information to Home Assistant.
[12:09:36] INFO: Successfully send service information to the Supervisor.

Anything in the add-on logs that might be useful for us?

No response

Additional information

/mnt/data/supervisor/addons/data/core_mosquitto/options.json

{
  "logins": [],
  "require_certificate": true,
  "certfile": "fullchain.pem",
  "keyfile": "privkey.pem",
  "customize": {
    "active": true,
    "folder": "mosquitto"
  }
}

mosquitto/mqtt.conf

#log_timestamp_format %Y-%m-%d %H:%M:%S %Z
doronazl commented 9 months ago

same here, no ones on this issue ?

homeerik commented 9 months ago

After upgrading to OS 11 i have random stops of Zigbee. Sometime al is working fine for a few days en suddenly all Zigbee devices are unavailable. I can restart HA anymore and only remove power. After reboot al is working fine until it is happening again.

Before the upgrade Zigbee and HA was rock solid.

I use the combination: Sonoff 3.0 P version usb stick Mosquitto broker Zigbee2MQTT

Log files are not showing anything.

blacknell commented 9 months ago

I could not wait long for a reply. This meant I had to remove and install mosquitto.

inonoob commented 9 months ago

Dear all,

I also have the same issue. I've remove at least 3 time mosquitto and re-installed it. That fixes the issue for some time but then it starts again with the same error message:

[12:14:24] INFO: Successfully send service information to the Supervisor. [10:14:24] INFO: Service restart after closing [12:14:24] INFO: Successfully send discovery information to Home Assistant. [12:14:25] INFO: Starting mosquitto MQTT broker... 2023-10-20 12:14:25: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persist__restore_sub: Assertion "client_id" failed. [12:14:26] INFO: Successfully send service information to the Supervisor. [10:14:26] INFO: Service restart after closing

Something seems wrong.

Home assistant OS 11.0 and

doronazl commented 9 months ago

Circling back just to share what i did. I have a lot of important sensors relying on the stability of mqtt, cant have this kind of unplanned problems and sit around waiting for an update again.

What i did is reverted to the backup before the update and went the route of seperating the mqtt from home assistant. All you need is a raspberry pi 3, installed mosquitto mqtt on it, changed settings to route to the new server and uninstalled the mosquito addon, now no update from home assistant cant effect my mqtt server uptime, i then updated the OS and moved on from this problem. This may not be a fix but a workaround, but its also making my system more resilient, happy with the results

inonoob commented 9 months ago

@doronazl

This is what I'm doing. I move now MQTT to a virtual debian machine with influxdb and Grafana with Promox. I don't trust the add-ons anymore. After an update in home assistant. The system just remove all my data I had collected in influxdb (3month in total lost). Since then I move now all to a dedicated machine for that task. And Home assistant is also going to move from the Pi to the Virtual machine.

doronazl commented 9 months ago

@doronazl

This is what I'm doing. I move now MQTT to a virtual debian machine with influxdb and Grafana with Promox. I don't trust the add-ons anymore. After an update in home assistant. The system just remove all my data I had collected in influxdb (3month in total lost). Since then I move now all to a dedicated machine for that task. And Home assistant is also going to move from the Pi to the Virtual machine.

Your way probably better, im just not a fan of VMs unless you know exactly what youre doing, otherwise lots of simple things can get complicated later when having to deal with those layers.

Im old fashioned, my HA is on a NUC, my mqtt on a rpi, they are both battery backed to keep things running at all times

inonoob commented 9 months ago

It might not be the best way but I had a server laying around and found a good use for it.

And for the authors of the add-on. I still would like to say thank you. I know open source can be sometimes difficult. Doing stuff in the spear time. So big thank you again. I learnt a lot with it.

DarthSonic commented 9 months ago

Solution is to uninstall and the re-install mosquitto addon. Configuration and everything is still intact after that.

blacknell commented 9 months ago

Solution is to uninstall and the re-install mosquitto addon. Configuration and everything is still intact after that.

This wasn't my experience. Uninstalling meant I lost the persistent mosquito database and so all retained messages.

I lost a lot of devices since the discovery was deleted. I'm not sure why the devices were deleted since the discovery payloads weren't cleared but that's what happened.

DarthSonic commented 9 months ago

Solution is to uninstall and the re-install mosquitto addon. Configuration and everything is still intact after that.

This wasn't my experience. Uninstalling meant I lost the persistent mosquito database and so all retained messages.

I lost a lot of devices since the discovery was deleted. I'm not sure why the devices were deleted since the discovery payloads weren't cleared but that's what happened.

Did not notice that, as I do not need any persistent currently.

kazikskrz commented 9 months ago

Is there a quick fix available without reinstalling?

skank01 commented 9 months ago

Same problem here

jt-nti commented 8 months ago

If it's any help, I managed to get slightly further with a mosquitto.conf file (using the customize flag) and setting persistence to false. I think that's causing its own issues but at least Zigbee2MQTT is connecting now.

rootiam commented 8 months ago

If it's any help, I managed to get slightly further with a mosquitto.conf file (using the customize flag) and setting persistence to false. I think that's causing its own issues but at least Zigbee2MQTT is connecting now.

I have the same issue. This didn't work for me.

jt-nti commented 8 months ago

I've had a chance to get hold of the persistance db out of the mosquitto addon and have a look using the mosquitto_db_dump tool, and there are some null client IDs in there which kind of makes sense based on the log...

DB_CHUNK_SUB:
    Length: 38
    Client ID: (null)
    Topic: stat/tasmota_000000/RESULT
    QoS: 0
    Subscription ID: 0
    Options: 0x00

I wonder if there's a mosquitto issue letting null client IDs through but then failing to process them? They were all tasmota related in my db, so that seems worth a look as well.

hackex commented 8 months ago

I've got into the same issue of error in mosquitto addon: "persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.[". Sadly, none of my mosquitto messages get received because of this error. I hope for a fix.

CaptainBalou commented 8 months ago

Same here.

[00:26:21] INFO: Starting mosquitto MQTT broker... 2023-11-18 00:26:21: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persist__restore_sub: Assertionclient_id' failed. [23:26:22] INFO: Service restart after closing [00:26:22] INFO: Successfully send discovery information to Home Assistant. [00:26:22] INFO: Successfully send service information to the Supervisor. [00:26:23] INFO: Starting mosquitto MQTT broker... 2023-11-18 00:26:23: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persistrestore_sub: Assertion client_id' failed. [23:26:23] INFO: Service restart after closing [00:26:23] INFO: Successfully send discovery information to Home Assistant. [00:26:24] INFO: Successfully send service information to the Supervisor. [00:26:24] INFO: Starting mosquitto MQTT broker... 2023-11-18 00:26:24: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persist__restore_sub: Assertionclient_id' failed. [23:26:24] INFO: Service restart after closing [00:26:25] INFO: Successfully send discovery information to Home Assistant. [00:26:25] INFO: Starting mosquitto MQTT broker... 2023-11-18 00:26:25: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persistrestore_sub: Assertion client_id' failed. [00:26:25] INFO: Successfully send service information to the Supervisor. [23:26:26] INFO: Service restart after closing [00:26:26] INFO: Successfully send discovery information to Home Assistant. [00:26:27] INFO: Starting mosquitto MQTT broker... 2023-11-18 00:26:27: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persist__restore_sub: Assertionclient_id' failed. [00:26:27] INFO: Successfully send service information to the Supervisor. [23:26:27] INFO: Service restart after closing [00:26:28] INFO: Successfully send discovery information to Home Assistant. [00:26:28] INFO: Starting mosquitto MQTT broker... 2023-11-18 00:26:28: Warning: Mosquitto should not be run as root/administrator. mosquitto: persist_read.c:550: persist__restore_sub: Assertion client_id' failed. [00:26:28] INFO: Successfully send service information to the Supervisor. [23:26:29] INFO: Service restart after closing [00:26:29] INFO: Successfully send discovery information to Home Assistant. [00:26:30] INFO: Starting mosquitto MQTT broker... . . .

fsaris commented 8 months ago

Same issue here. After reinstall it worked again (for now).

jt-nti commented 8 months ago

@hackex @CaptainBalou @fsaris do you have any Tasmota devices? That seems to be what caused the problem for me- at least everything has been working since fixing the mosquitto.db and removing the problematic Tasmota device. If not, it would be interesting to see which client has the null id in your case. Unfortunately it's a bit fiddly to find that out, so it might be nice to update the mosquitto addon to (optionally?) log the output of a db_dump at start up?

fsaris commented 8 months ago

Yes, I have tasmota devices.

So you expect that one of the devices broke the mosquitto db?

stfnhmplr commented 8 months ago

Same here. I have also Tasmota devices. 

2023-11-19 10:50:17: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[09:50:17] INFO: Service restart after closing
[10:50:18] INFO: Starting mosquitto MQTT broker...
2023-11-19 10:50:18: Warning: Mosquitto should not be run as root/administrator.
mosquitto: persist_read.c:550: persist__restore_sub: Assertion `client_id' failed.
[09:50:18] INFO: Service restart after closing
[10:50:19] INFO: Starting mosquitto MQTT broker...
jt-nti commented 8 months ago

@fsaris it's a possibility. If you're able to build and run the db_dump tool, it should show the null client ID. (I still have three Tasmotas which aren't causing any problems so I'm not sure why one of them had the null client ID.)

inonoob commented 8 months ago

I can give the feedback I don't have Tasmota devices but I have that issue. I have AI on the Edge. I don't know if that might be a problem aswell. So it might not only be for one device brand only.

obfri commented 8 months ago

Same problem detected today - even with MQTT 6.4.0. Using Tasmota devices.

obfri commented 8 months ago

my tasmota devices showing MQTT login errors - nothing changed: 20:52:16.971 MQT: Attempting connection... 20:52:16.988 MQT: Connect failed to 192.168.xxx.xxx:1883, rc -2. Retry in 20 sec 20:52:37.964 MQT: Attempting connection... 20:52:37.985 MQT: Connect failed to 192.168.xxx.xxx:1883, rc -2. Retry in 30 sec

did some more investigations: failure started after MQTT 6.4.0 update (done Nov 20) full restore of Nov19 full backup eliminates failure

jt-nti commented 7 months ago

The good news is that there's a fix coming soon (hopefully 🤞) from mosquitto which should help...

https://github.com/eclipse/mosquitto/commit/fd097d5a0f4d26f5d5ae35cf1fb326c373ad39e1

I still think it would be nice if the home assistant addon could include the mosquitto_db_dump tool as well- not really sure where to start but I might attempt to have a look over the Christmas break.

github-actions[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jt-nti commented 6 months ago

It would be good to use the updated version of mosquitto when it's available, and including the db_dump tool might still be nice to help debug any future problems.

github-actions[bot] commented 5 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jt-nti commented 5 months ago

It doesn't look like the mosquitto fix has been released yet.

github-actions[bot] commented 4 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

palsch commented 3 weeks ago

It looks like there is something wrong with the HA addon that didn't send the client ID to Mosquitto. This leads to broken messages that were persisted by Mosquitto. After a restart, Mosquitto tries to load the persisted messages and fails because of the missing client ID in those messages. To fix this, you have to delete the persisted messages with the following command (! removes the complete mosquitto DB with all persisted messages):

sudo rm /var/lib/mosquitto/mosquitto.db
RosemaryOrchard commented 3 weeks ago

To fix this, you have to delete the persisted messages with the following command:

sudo rm /var/lib/mosquitto/mosquitto.db

For anyone not familiar with terminal commands: This deletes the whole mosquitto DB. The problem messages could be removed from the DB instead of removing everything.

palsch commented 3 weeks ago

To fix this, you have to delete the persisted messages with the following command:

sudo rm /var/lib/mosquitto/mosquitto.db

For anyone not familiar with terminal commands: This deletes the whole mosquitto DB. The problem messages could be removed from the DB instead of removing everything.

Thats true, if you know how to open the db file, find the broken message and how to remove this.

Removing the full db is simpler. And it is simpler than trying to reistall the complete mosquitto and maybe have to reconfigure that.

jt-nti commented 3 weeks ago

It would be really nice to fix this issue, unfortunately it got closed as stale.

There is a fix in mosquitto that will fix the issue with missing client IDs in the persistence file but I don't think it's in a release yet- that's how someone fixed my file rather than deleting it, although deleting it is probably way easier!

It would also be good if nothing was sent without a client ID in the first place.