Zigbee2MQTT doesn't start after Update to Core 2024.1.0

Maggan0 commented 11 months ago

Description of the issue

Zigbee2MQTT doesn't start after Update to Core 2024.1.0.

Addon version

1.35.0-1

Platform

Core 2024.1.0 Supervisor 2023.12.0 Operating System 11.3 Frontend 20240103.3

Logs of the issue (if applicable)

[13:29:31] INFO: Preparing to start... [13:29:31] INFO: Socat not enabled [13:29:32] INFO: Starting Zigbee2MQTT... Zigbee2MQTT:info 2024-01-05 13:29:34: Logging to console and directory: '/config/zigbee2mqtt/log/2024-01-05.13-29-34' filename: log.txt Zigbee2MQTT:info 2024-01-05 13:29:34: Starting Zigbee2MQTT version 1.35.0 (commit #unknown) Zigbee2MQTT:info 2024-01-05 13:29:34: Starting zigbee-herdsman (0.30.0) Assertion failed: Command (setConfigurationValue) returned unexpected state: [object Object] Assertion failed: Command (setValue) returned unexpected state: 55 Error: {"address":0,"clusterId":32770,"sequence":2} after 10000ms at Timeout._onTimeout (/app/node_modules/zigbee-herdsman/src/utils/waitress.ts:64:35) at listOnTimeout (node:internal/timers:569:17) at processTimers (node:internal/timers:512:7)

m4dm4xi commented 11 months ago

Same for me.

In my case it goes hand in hand with the update of: Operating System 11.2 -> 11.3 Silabs Multiprotocol Addon 2.3.0 -> 2.4.0

The problem already appeared before I afterwards updated Core from 2023.12 -> 2024.1

Same log:

[13:49:13] INFO: Preparing to start... [13:49:14] INFO: Socat not enabled [13:49:17] INFO: Starting Zigbee2MQTT... Zigbee2MQTT:info 2024-01-05 13:49:24: Logging to console and directory: '/config/zigbee2mqtt/log/2024-01-05.13-49-24' filename: log.txt Zigbee2MQTT:info 2024-01-05 13:49:24: Starting Zigbee2MQTT version 1.35.0 (commit #213 Zigbee2MQTT:info 2024-01-05 13:49:24: Starting zigbee-herdsman (0.30.0) Assertion failed: Command (setConfigurationValue) returned unexpected state: [object Object] Assertion failed: Command (setValue) returned unexpected state: 55 Error: {"address":0,"clusterId":32770,"sequence":2} after 10000ms at Timeout._onTimeout (/app/node_modules/zigbee-herdsman/src/utils/waitress.ts:64:35) at listOnTimeout (node:internal/timers:569:17) at processTimers (node:internal/timers:512:7)

Maggan0 commented 11 months ago

I updated both at the same time. Possible that this issue comes from the OS.

rafpigna commented 11 months ago

+1 here, same error Like the user before, I updated Core, OS and Silab addon before Z2M How much I hate this stuff when updtating software, luckly I just have the backup to restore, but I will wait a couple of hours, I dont want to make a mess...

Anyway my error log has something different from the ones reported above, I see a different row on mine, the ones that starts with " Using zigbee-herdsman with settings:......"

[15:27:19] INFO: Preparing to start...
[15:27:19] INFO: Socat not enabled
[15:27:20] INFO: Starting Zigbee2MQTT...
Zigbee2MQTT:debug 2024-01-05 15:27:21: Loaded state from file /config/zigbee2mqtt/state.json
Zigbee2MQTT:info  2024-01-05 15:27:21: Logging to console and directory: '/config/zigbee2mqtt/log/2024-01-05.15-27-21' filename: log.txt
Zigbee2MQTT:debug 2024-01-05 15:27:21: Removing old log directory '/config/zigbee2mqtt/log/2024-01-05.15-22-48'
Zigbee2MQTT:info  2024-01-05 15:27:21: Starting Zigbee2MQTT version 1.35.0 (commit #unknown)
Zigbee2MQTT:info  2024-01-05 15:27:21: Starting zigbee-herdsman (0.30.0)
Zigbee2MQTT:debug 2024-01-05 15:27:21: Using zigbee-herdsman with settings: '{"adapter":"concurrent":null,
"delay":null,"disableLED":false},"backupPath":"/config/zigbee2mqtt/coordinator_backup.json",
"databaseBackupPath":"/config/zigbee2mqtt/database.db.backup","databasePath":"/config/zigbee2mqtt/database.db",
"network":{"channelList":[11],"extendedPanID":61,10,3,241,255,67,153,169],"networkKey":"HIDDEN",
"panID":38409},"serialPort":{"adapter":"ezsp","path":"tcp://core-silabs-multiprotocol:9999"}}'
Assertion failed: Command (setConfigurationValue) returned unexpected state: [object Object]
Assertion failed: Command (setValue) returned unexpected state: 55
Error: {"address":0,"clusterId":32770,"sequence":2} after 10000ms
    at Timeout._onTimeout (/app/node_modules/zigbee-herdsman/src/utils/waitress.ts:64:35)
    at listOnTimeout (node:internal/timers:569:17)
    at processTimers (node:internal/timers:512:7)

NewsGuyTor commented 11 months ago

Yeah, I upgraded only the OS today (upgraded to HA 2024 yesterday), and that's what broke Z2M.

NewsGuyTor commented 11 months ago

ha os update --version 11.2 rolls the OS back to 11.2 quickly!

Still can't get Z2M to work though, perhaps it is the Silicon Labs Multiprotocol 2.4.0 that is the issue.

NewsGuyTor commented 11 months ago

Confirmed, got it working again now! Silicon Labs Multiprotocol 2.4.0 is the issue! Go to Settings - Backups and restore addon_core_silabs_multiprotocol_2.3.2

rafpigna commented 11 months ago

Confirmed, got it working again now! Silicon Labs Multiprotocol 2.4.0 is the issue! Go to Settings - Backups and restore addon_core_silabs_multiprotocol_2.3.2

uhm... this is strange. just reverted to silab 2.3.2 and I still got an error starting z2m, but different

Zigbee2MQTT:error 2024-01-05 15:47:28: Error while starting zigbee-herdsman Zigbee2MQTT:error 2024-01-05 15:47:28: Failed to start zigbee Zigbee2MQTT:error 2024-01-05 15:47:28: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions Zigbee2MQTT:error 2024-01-05 15:47:28: Exiting... Zigbee2MQTT:error 2024-01-05 15:47:28: Error: Failure send version:{"type":"Buffer","data":[0,0,0,4]} at /app/node_modules/zigbee-herdsman/src/adapter/ezsp/driver/ezsp.ts:562:23 at processTicksAndRejections (node:internal/process/task_queues:95:5) at Queue.executeNext (/app/node_modules/zigbee-herdsman/src/utils/queue.ts:32:32)

anyway after a reboot it starts again, but all my zigbee devices are reported as offline.

rafpigna commented 11 months ago

I dont understand why, but after stopping and rebooting just the Silab addon, the devices come back online and now all works.

So yes, seems that updating to Silicon Labs Multiprotocol 2.4.0 cause Zigbee2Mqtt to not start anymore.

1PlusN commented 11 months ago

I dont understand why, but after stopping and rebooting just the Silab addon, the devices come back online and now all works.

So yes, seems that updating to Silicon Labs Multiprotocol 2.4.0 cause Zigbee2Mqtt to not start anymore.

+1 confirming 2.4.0 seems to be the issue. backup restore plus HA reboot seems to have fixed the starting issue. now getting this error on Z2M

Error 2024-01-05 10:19:38Publish 'set' 'state' to 'Front Door Light' failed: 'Error: Command 0x040d84fffed8e589/1 genOnOff.on({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Timeout - 27084 - 1 - 6 - 6 - 11 after 10000ms)'

Edit: after a full reboot of the vm the error seems to have resolved. HA restarts did not resolve.

n-prat commented 11 months ago

Various upgrades

cf error from OP

Tried to restore separately OS,Core,Z2M,SiLabs but no luck.

RESTORE OLD

OS: 11.2 Core: 2023.12.4 Silicon Labs Multiprotocol: 2.3.2 Zigbee2MQTT: 1.34.0-1

Without unpluging: all "offline" Even with STOP+START SiLabs addon (Maybe could have fixed itself by waiting?)

Unplug/replug all lamps, and press on remotes --> OK

NEW

Upgrade:
- Zigbee2MQTT
- OS
- Core (in this order)
Reboot machine ("host reboot")

-----------------------------------------------------------
 Add-on version: 2.3.2
 There is an update available for this add-on!
 Latest add-on version: 2.4.0
 Please consider upgrading as soon as possible.
 System: Home Assistant OS 11.3  (amd64 / qemux86-64)
 Home Assistant Core: 2024.1.0
 Home Assistant Supervisor: 2023.12.0
--------------------------------------------------------

--> OK

ncodee commented 11 months ago

Reverting back to Silicon Labs Multiprotocol 2.3.2 from 2.40, with the HA OS 11.3 update seems to have fixed the Zigbee2MQTT.

What a shitshow of a release. The amount of bugs and issues with the new HA 2024.1 release is outstanding, it's clear it was a rushed release with little or no testing at the beta stage.

kerfich commented 11 months ago

Same situation here I lost communication between SiliconLabs Multliprotocol and Zigbee2MQTT after the 2.4.0 upgrade. Everything is fine now after reverting to 2.3.2 HAOS on VM Core: 2024.1.0 Supervisor: 2023.12.0 Operating System: 11.3

velodromen commented 11 months ago

same problem here after updating SiliconLabs Multliprotocol to 2.4.0 6 hours ago (previously updated to 2024.1.0 without issue). I had by this time not updated OS to 11.3.

I have now updated to OS 11.3 and restored SiliconLabs Multliprotocol to 2.3.2. This worked once I rebooted and waited for Zigbee2MQTT to fully load.

Thank you for finding the root cause and publishing it here! Absolutely love the HA community, you're great!

agners commented 11 months ago

Since also @Maggan0 reported that he updated to the latest Silabs Multiprotocol Addon at the same time, this most likely related purely to the Silabs Multiprotocol Addon update.

Typically, we mostly focus our tests on ZHA. The EmberZNet protocol is relatively stable, so far we never had a breakage which was only specific to Z2M. I guess this marks the first time :cry:

I don't see anything which stands out in the Zigbee EmberZNet 7.4.0.0 release notes. The protocol version got updated to 0x0D (13), mayeb this is the problem?

@kirovilya maybe you can chime in here?

agners commented 11 months ago

What a shitshow of a release. The amount of bugs and issues with the new HA 2024.1 release is outstanding, it's clear it was a rushed release with little or no testing at the beta stage.

Keep in mind that the Silicon Labs Multiprotocol add-on is marked as experimental still. As well as EmberZNet support on Z2M side. I don't really understand your comment here, especially since the fix is a simple restore of the Silicon Labs Multiprotocol add-on to revert to the previous release.

nic0dk commented 11 months ago

restore of the Silicon Labs Multiprotocol add-on to revert to the previous release.

sorry for asking - do you know how to downgrade ? i simply cant figure out where to look.

NewsGuyTor commented 11 months ago

restore of the Silicon Labs Multiprotocol add-on to revert to the previous release.

sorry for asking - do you know how to downgrade ? i simply cant figure out where to look.

https://github.com/zigbee2mqtt/hassio-zigbee2mqtt/issues/555#issuecomment-1878792906

rafpigna commented 11 months ago

restore of the Silicon Labs Multiprotocol add-on to revert to the previous release.

sorry for asking - do you know how to downgrade ? i simply cant figure out where to look.

restore a backup (choosing partial restore and then only the addon) is the best way. If you dont have a recent backup (always, always, always! take a backup before updating anything, even the most "stupid" addon) you can unistall the addon and install again but you have to find the older version. Before uninstalling you can save the addon configuration file yaml and copy it in the new installation.

nic0dk commented 11 months ago

THX after downgrading i got it working

puddly commented 11 months ago

I strongly suggest you migrate away from multi-protocol.

Z2M's support for the EZSP protocol is still experimental and the multi-PAN addon will always be using the absolute latest release of the SDK. Get a second stick for Thread or just reinstall normal EmberZNet firmware if you aren't using Thread. Otherwise, it will continue to break every time the addon is updated.

NewsGuyTor commented 11 months ago

Get a second stick for Thread or just reinstall normal EmberZNet firmware if you aren't using Thread. Otherwise, it will continue to break every time the addon is updated.

That doesn't make sense logically, considering this is the first time in years it breaks.

I'd bet most would rather spend minutes to roll back any broken updates than spending hours re-pairing their whole zigbee network to avoid spending minutes on potential rollbacks.

puddly commented 11 months ago

That doesn't make sense logically, considering this is the first time in years it breaks.

The addon was released a little over a year ago and your setup will break once again when a new version of the addon is published in a few days. It's not an issue with HA Core, HA OS, or the addon itself. Disable auto-updates if you plan on continuing to use it.

than spending hours re-pairing their whole zigbee network to avoid spending minutes on potential rollbacks.

You can migrate from multi-PAN back to Zigbee-only without losing any settings:

pip install zigpy-cli
zigpy radio --baudrate 115200 ezsp socket://core-silabs-multiprotocol:9999 backup -z > backup.json
# Now, stop the multi-protocol addon
# Flash new firmware (https://skyconnect.home-assistant.io/firmware-update/ or https://darkxst.github.io/silabs-firmware-builder/)
# Restore the backup
zigpy radio --baudrate 115200 ezsp /dev/serial/by-id/your-stick restore backup.json

Update Z2M to use the correct serial port and you should be back up and running.

NewsGuyTor commented 11 months ago

Disable auto-updates if you plan on continuing to use it.

Already done ;)

You can migrate from multi-PAN back to Zigbee-only without losing any settings:

Thanks, that was easier than expected. Is the same possible for Thread?

Even if turns out to be easy too, I still think it is strangely wasteful that two separate identical hardware devices are recommended, only to avoid issues with software updates, not software/hardware issues.

It seems like it's a much better and less waste-producing solution to change the software update process to avoid future similar issues rather than encouraging such superfluous CO2 to avoid issues that are caused by how updates are tested/released.

Perhaps the current Silicon Labs Multiprotocol add-on should be considered beta/stable and the troublesome process of always using the absolute latest release of the SDK without testing for issues should be reserved for a new Silicon Labs Multiprotocol Alpha add-on?

I think that would be a much better solution than encouraging hardware purchases to avoid software update issues.

CommanderROR9 commented 11 months ago

I ran into this issue as well...after rolling back to the older multiprotocol addon Z2MQTT is loading again, but all my lights are still showing as "Offline" and can't be controlled.

1PlusN commented 11 months ago

Anyone tried 2.4.2 yet? Don’t have time to debug this evening.

Maggan0 commented 11 months ago

Yes. After the new updates everything seems to be fine.

CommanderROR9 commented 11 months ago

Followup on my end... rolling back (later Updating to 2.4.1) did not immediately solve everything. I had to manually power cycle all my Zigbee Lights for full functionality to be restored.

Koenkk commented 11 months ago

@puddly should 2.4.2 work fine? If yes, why would it break again with a new update?

The addon was released a little over a year ago and your setup will break once again when a new version of the addon is published in a few days.

Maggan0 commented 11 months ago

I want to do the rollback today, but saw the new update yesterday in the night and just give it a try. Everything works fine, but I will check it properly later today.

puddly commented 11 months ago

@Koenkk 2.4.2 works fine because it's a revert of the addon to exactly the way it was a few days ago. It'll break again the moment 2.4.3 is deployed, which is again built with Gecko SDK 4.4.0.

@NewsGuyTor The multiprotocol addon is marked Experimental because it's not close to alpha or beta quality. Working != stable!

the troublesome process of always using the absolute latest release of the SDK without testing for issues

This addon release is very much the result of months of testing and bugfixing. It fixes critical bugs affecting people using multiprotocol for heavy concurrent Zigbee and Thread+Matter. I've been running it for weeks now.

Please remember that Z2M's support for EZSP is itself experimental:

The adapters below are experimental, don't use these if you want a stable setup. Based on Silicon Labs EFR32MG2x/MGM21x and EFR32MG1x/MGM1x series

When you use an experimental addon running experimental firmware on an adapter with experimental support, expect breakage. Z2M's support for EZSP is still in-progress. If you want stability, migrate to the well-supported setup of two separate sticks: one for Thread and one for Zigbee.

Koenkk commented 11 months ago

@puddly does this summarise it correctly?

People shouldn't use the experimental HA multiprotocol addon
No changes from the zigbee2mqtt side are required

CommanderROR9 commented 11 months ago

That's all find and dandy, but in order to use Matter/Thread with the SkyConnect Stick (which was advertised for exactly this Usecase) there is no other choice than to use the addon. If all this is so experimental, then why not offer a stable baseline version and put all the trial and error stuff in an "edge" version? Currently both ZHA and Zigbee2mqtt are suffering catastrophic failures after having run fine for many months. This is fine for a dev build, but not so fine if it's the only available build in Home Assistant.

joe-sydney commented 11 months ago

I rolled back to 2.3.2 (via backup & restore). Is there any reason to upgrade to 2.4.2 (eg there are other improvements other than Gecko SDK version update)?

2.3.2 has been stable for me, are there any critical issues that need an upgrade, especially security related? Otherwise, I would be happy to stay with what I have till things get sorted. I presume Z2M needs some updates to become compatible with the new Gecko SDK version?

In the meantime, as suggested in the above post, perhaps a stable version that works for Z2M (and ZHA) can be maintained with any critical fixes applied as needed.

puddly commented 11 months ago

@Koenkk Not quite. Z2M doesn't support any EZSP stick running Gecko SDK 4.4.0/EmberZNet 7.4.0. The multiprotocol addon exposes a virtual adapter over TCP so it behaves no different, it just happened to run into this issue because its firmware auto-updates.

Koenkk commented 11 months ago

@puddly thanks!

Tracking the support of 7.4.0 in https://github.com/Koenkk/zigbee-herdsman/issues/854

zigbee2mqtt / hassio-zigbee2mqtt