Entire Zigbee network vanishing after reboot or touchlink

max-rousseau commented 2 years ago

What happened?

Hello All - Hoping someone can help out with this pretty catastrophic failure scenario. Let me provide some background / context to help reflect on troubleshooting:

Background

I'm running HASSIO with version 2022.4.7 of home assistant
The zigbee2mqtt service is installed as a supervisor add-on. Currently v1.25.1-1.
The controller hardware I'm using is a sonoff zigbee 3.0 plus usb stick. I have flashed it with the latest version firmware and saw as much in the logs: 2022-04-26 00:00:21: Coordinator firmware version: '{"meta":{"maintrel":1,"majorrel":2,"minorrel":7,"product":1,"revision":20220219,"transportrev":2},"type":"zStack3x0"}'
I have double-checked that the USB device defined is correct and in fact able to communicate (log item above is good indicator also).
My network has 30 devices, a mix of Hue lights, Ikea Wall Switches and Aqara sensors. None of them works now.

Example failure log:

error 2022-04-26 10:13:16: Publish 'set' 'state' to 'Hue Ambiance Light 1' failed: 'Error: Command 0x001788010b5da159/11 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Data request failed with error: 'No network route' (205))'

In the mqtt topic, all devices have linkquality:null

What I've tried so far:

Made sure firmware was updated
Rebooted hardware
Restarted HA, z2m, mqtt - all and any order really

Added the following to HA config:

birth_message:
topic: "homeassistant/status"
payload: "online"
will_message:
topic: "homeassistant/status"
payload: "offline"

What caused this?

First time, this happened after a powerloss to the house
Second time, appears to have happened right after using touchlink to reset a Hue Color bulb

I'm kind of at my wits end now. Could really use a hand

Originally posted by @max-rousseau in https://github.com/Koenkk/zigbee2mqtt/discussions/12289

What did you expect to happen?

Anything else than zigbee armageddon :)

How to reproduce it (minimal and precise)

Hard to say - suspecting touchlink can reproduce but hard to test.

Zigbee2MQTT version

1.25.1-1

Adapter firmware version

20220219

Adapter

Sonoff Zigbee USB Stick 3.0 Plus

Debug log

debug 2022-05-09 08:03:08: Loaded state from file /config/zigbee2mqtt/state.json info 2022-05-09 08:03:08: Logging to console and directory: '/config/zigbee2mqtt/log/2022-05-09.08-03-08' filename: log.txt debug 2022-05-09 08:03:08: Removing old log directory '/config/zigbee2mqtt/log/2022-05-08.21-55-42' info 2022-05-09 08:03:08: Starting Zigbee2MQTT version 1.25.1 (commit #unknown) info 2022-05-09 08:03:08: Starting zigbee-herdsman (0.14.27) debug 2022-05-09 08:03:08: Using zigbee-herdsman with settings: '{"adapter":{"concurrent":null,"delay":null,"disableLED":false},"backupPath":"/config/zigbee2mqtt/coordinator_backup.json","databaseBackupPath":"/config/zigbee2mqtt/database.db.backup","databasePath":"/config/zigbee2mqtt/database.db","network":{"channelList":[25],"extendedPanID":[221,221,221,221,221,221,221,221],"networkKey":"HIDDEN","panID":6754},"serialPort":{"adapter":"zstack","path":"/dev/ttyUSB0"}}' info 2022-05-09 08:03:09: zigbee-herdsman started (resumed) info 2022-05-09 08:03:09: Coordinator firmware version: '{"meta":{"maintrel":1,"majorrel":2,"minorrel":7,"product":1,"revision":20220219,"transportrev":2},"type":"zStack3x0"}' debug 2022-05-09 08:03:09: Zigbee network parameters: {"channel":25,"extendedPanID":"0x00124b0024c911e0","panID":6754} info 2022-05-09 08:03:09: Currently 28 devices are joined: info 2022-05-09 08:03:09: Hue Module Blue (0x001788010b02ae89): 929003017102 - Philips Hue wall switch module (EndDevice) info 2022-05-09 08:03:09: Hue White Par38 A (0x00178801045b9953): 046677476816 - Philips Hue white PAR38 outdoor (Router) info 2022-05-09 08:03:09: Hue White Par38 B (0x00178801045b998c): 046677476816 - Philips Hue white PAR38 outdoor (Router)

[snipped]

warn 2022-05-09 08:03:09: permit_join set to true in configuration.yaml. warn 2022-05-09 08:03:09: Allowing new devices to join. warn 2022-05-09 08:03:09: Set permit_join to false once you joined all devices. info 2022-05-09 08:03:09: Zigbee: allowing new devices to join. info 2022-05-09 08:03:09: Connecting to MQTT server at mqtt://(redacted):1883 debug 2022-05-09 08:03:09: Using MQTT keepalive: 60 debug 2022-05-09 08:03:09: Using MQTT login with username: foo info 2022-05-09 08:03:09: Connected to MQTT server info 2022-05-09 08:03:09: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload '{"state":"online"}' debug 2022-05-09 08:03:09: Received MQTT message on 'zigbee2mqtt/bridge/info' with data '{"commit":"unknown","config":{"advanced":{"adapter_concurrent":null,"adapter_delay":null,"availability_blacklist":[],"availability_blocklist":[],"availability_passlist":[],"availability_whitelist":[],"cache_state":true,"cache_state_persistent":true,"cache_state_send_on_startup":true,"channel":25,"elapsed":false,"ext_pan_id":[221,221,221,221,221,221,221,221],"homeassistant_legacy_entity_attributes":false,"last_seen":"disable","legacy_api":false,"legacy_availability_payload":false,"log_directory":"/config/zigbee2mqtt/log/%TIMESTAMP%","log_file":"log.txt","log_level":"info","log_output":["console","file"],"log_rotation":true,"log_symlink_current":false,"log_syslog":{"app_name":"Zigbee2MQTT","eol":"/n","host":"localhost","localhost":"localhost","path":"/dev/log","pid":"process.pid","port":123,"protocol":"tcp4","type":"5424"},"output":"json","pan_id":6754,"report":false,"soft_reset_timeout":0,"timestamp_format":"YYYY-MM-DD HH:mm:ss"},"blocklist":[],"device_options":{"legacy":false},"devices":{"0x0000000000402fda":{"friendly_name":"Hue Tap Red"},"0x0000000000428522":{"friendly_name":"Hue Tap Alpha"},"0x0000000000459fbc":{"friendly_name":"Hue Tap Purple"},"0x00124b0025107f49":{"friendly_name":"Sonoff Temperature Green"},"0x00124b00251e27e1":{"friendly_name":"Sonoff Door Green"},"0x00158d0007b06ce1":{"friendly_name":"Aqara Door Sensor 1"},"0x00158d000806cba2":{"friendly_name":"Aqara Door Blue"},"0x0017880100b63abc":{"friendly_name":"Hallway 2"},"0x0017880100b6433f":{"friendly_name":"Hallway 1"},"0x0017880100bfde17":{"friendly_name":"Hallway 3"},"0x001788010202f5cc":{"friendly_name":"Hue Motion Blue"},"0x0017880102031bdd":{"friendly_name":"Hue Motion Pink"},"0x00178801021196c4":{"friendly_name":"Hue Motion Purple"},"0x001788010211d1a8":{"friendly_name":"Hue Motion Red"},"0x00178801045b9953":{"friendly_name":"Hue White Par38 A"},"0x00178801045b998c":{"friendly_name":"Hue White Par38 B"},"0x001788010645fd72":{"friendly_name":"Hue Outdoor Motion Mailbox"},"0x001788010804691b":{"friendly_name":"Hue Smart Plug Purple"},"0x00178801091983b4":{"friendly_name":"Hue Motion Green"},"0x001788010b02ae89":{"friendly_name":"Hue Module Blue"},"0x001788010b5da159":{"friendly_name":"Hue Ambiance Light 1"},"0x001788010b92daa1":{"friendly_name":"Hue White Par38 D"},"0x001788010b92dab2":{"friendly_name":"Hue White Par38 C"},"0x00178801105e908b":{"friendly_name":"Hue Dimmer Purple"},"0x5c0272fffe61608d":{"friendly_name":"Ikea Plug Yellow"},"0x5c0272fffe7d7209":{"friendly_name":"Ikea Plug Blue"},"0x5c0272fffe7d7fdf":{"friendly_name":"Ikea Plug Purple"},"0x847127fffeae7e94":{"friendly_name":"Ikea Plug Pink"}},"external_converters":[],"frontend":{"host":"0.0.0.0","port":8099},"groups":{},"homeassistant":{"discovery_topic":"homeassistant","legacy_entity_attributes":false,"legacy_triggers":true,"status_topic":"hass/status"},"map_options":{"graphviz":{"colors":{"fill":{"coordinator":"#e04e5d","enddevice":"#fff8ce","router":"#4ea3e0"},"font":{"coordinator":"#ffffff","enddevice":"#000000","router":"#ffffff"},"line":{"active":"#009900","inactive":"#994444"}}}},"mqtt":{"base_topic":"zigbee2mqtt","force_disable_retain":false,"include_device_information":false,"keepalive":60,"server":"mqtt://mqtt.secured.org:1883","user":"foo","version":4},"ota":{"disable_automatic_update_check":false,"update_check_interval":1440},"passlist":[],"permit_join":false,"serial":{"adapter":"zstack","disable_led":false,"port":"/dev/ttyUSB0"}},"config_schema":{"definitions":{"device":{"properties":{"debounce":{"description":"Debounces messages of this device","title":"Debounce","type":"number"},"debounce_ignore":{"description":"Protects unique payload values of specified payload properties from overriding within debounce time","examples":["action"],"items":{"type":"string"},"title":"Ignore debounce","type":"array"},"filtered_attributes":{"description":"Filter attributes from publish payload.","examples":["temperature","battery","action"],"items":{"type":"string"},"title":"Filtered publish attributes","type":"array"},"filtered_optimistic":{"description":"Filter attributes from optimistic publish payload when calling /set. (This has no effect if optimistic is set to false).","examples":["color_mode","color_temp","color"]

[snipped]

debug 2022-05-09 09:22:12: Error: Command 0x001788010b5da159/11 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Data request failed with error: 'No network route' (205)) at ZStackAdapter.sendZclFrameToEndpointInternal (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/zStackAdapter.ts:414:23) at Queue.executeNext (/app/node_modules/zigbee-herdsman/src/utils/queue.ts:32:32) debug 2022-05-09 09:22:30: Received Zigbee message from 'Hue Motion Red', type 'attributeReport', cluster 'msIlluminanceMeasurement', data '{"measuredValue":17022}' from endpoint 2 with groupID 0

error 2022-05-09 09:24:35: Publish 'set' 'state' to 'Hue Ambiance Light 1' failed: 'Error: Command 0x001788010b5da159/11 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Data request failed with error: 'No network route' (205))' debug 2022-05-09 09:24:35: Error: Command 0x001788010b5da159/11 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Data request failed with error: 'No network route' (205)) at ZStackAdapter.sendZclFrameToEndpointInternal (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/zStackAdapter.ts:414:23) at Queue.executeNext (/app/node_modules/zigbee-herdsman/src/utils/queue.ts:32:32) debug 2022-05-09 09:25:57: Received MQTT message on 'zigbee2mqtt/Hue Ambiance Light 1/set' with data '{"state": "OFF"}' debug 2022-05-09 09:25:57: Publishing 'set' 'state' to 'Hue Ambiance Light 1' error 2022-05-09 09:26:19: Publish 'set' 'state' to 'Hue Ambiance Light 1' failed: 'Error: Command 0x001788010b5da159/11 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Data request failed with error: 'No network route' (205))' debug 2022-05-09 09:26:19: Error: Command 0x001788010b5da159/11 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Data request failed with error: 'No network route' (205)) at ZStackAdapter.sendZclFrameToEndpointInternal (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/zStackAdapter.ts:414:23) at Queue.executeNext (/app/node_modules/zigbee-herdsman/src/utils/queue.ts:32:32)

Koenkk commented 2 years ago

Before diving deep;

Can you confirm the sonoff dongle is connected via a USB extension cable?
If using an raspberry pi; can you confirm that the USB power source is good and that there are no other USB connected devices?
Could there be any interference( from e.g. WiFi) https://www.zigbee2mqtt.io/advanced/zigbee/02_improve_network_range_and_stability.html#usb-based-adapter

max-rousseau commented 2 years ago

Thanks for checking in @Koenkk -

Yes, the dongle is on a long extension to keep it away from interference (i also had not moved the dongle when things stopped working)
Not using a raspberry pi, this runs on hassio on a intel NUC. Its powered on its own power source.
Certainly possible that there would be, I did however design for zigbee to be on channel 25 and configured wifi to stay only on channels 1 and 6 for 2.4gz - based on this article: https://www.metageek.com/training/resources/zigbee-wifi-coexistence/

All the same - ill see if perhaps I can take a longer extender and move antenna elsewhere.

max-rousseau commented 2 years ago

I tried now a 10ft long USB extender to make sure the antenna was far away from any SSD/WIFI AP. Tried also replacing the screw-in antenna with a bigger one and also an omnidirectional one. Unfortunately that hasn't helped.

There has been 2-3 hue motion devices which reconnected (9290012607). There does not seem to be any logic to which ones, they aren't the closest ones to the antenna of the coordinator. The IKEA plugs (E1603/E1702/E1708) will reconnect and show LQI and be responsive if I push their reset button with a pin.

Nothing else so far.

max-rousseau commented 2 years ago

Three hue motion sensors (9290012607) have now also returned. But the original hue light bulb I was adding is no longer responding.

Koenkk commented 2 years ago

Any chance you could provide a sniff when controlling a bulb fails?

https://www.zigbee2mqtt.io/advanced/zigbee/04_sniff_zigbee_traffic.html#with-cc2531

max-rousseau commented 2 years ago

If its only possible with a CC2531 then I don't think so. I only have the sonoff zigbee 3.0 (CC2652P)and a Conbee 2

max-rousseau commented 2 years ago

Aright well few updates. I ordered a CC2531 because sniffing the traffic seems fun, should have it shortly. I have noticed some pretty odd behavior while trying to restore the network, I used Touchlink again to force reset one of my Hue Lights (046677476816). After I reset that bulb, using touchlink, i had a bunch of other hue lights cascade back into the network...

I am really thinking something about the touchlink is messing up the adapter. It's also often failing and i have to restart the addon. I probably have to try 3-4 times to successfully reset a light with touchlink.

netweaver1970 commented 2 years ago

Another testimonial, I only once tried the touchlink option, out of curiosity. I lost quite a few devices (maybe 6-7) on my network, I had to repair them all. I have an Sonoff USB stick connected to a 1m USB extention lead as coordinator. For now I'm not touching Touchlink with a pole anymore. Even more as I don't have a need for it, as I have no Hue devices. But something is indeed not kosher with that functionality on the Sonoff stick (or specific firmware).

My details:

Zigbee2MQTT version
1.25.0-dev commit: [f6ad3c9](https://github.com/Koenkk/zigbee2mqtt/commit/f6ad3c9)
Coordinator type
zStack3x0
Coordinator revision
20211217

max-rousseau commented 2 years ago

Similar here - I've also found that one can use a hue dimmer switch instead to factory reset the bulbs, that seems safer:

With the Philips Dimmer Switch in hand, bring it within 10 cm of the bulb or lamp you wish to factory reset, then press and hold the I and 0 buttons simultaneously. Continue holding for 10 seconds until the green LED illuminates on the dimmer switch, the bulb will flash just before receiving the green LED.

Koenkk commented 2 years ago

@max-rousseau when not using touchlink, does everything work fine? There were some touchlink issues before which Texas Instruments should have fixed in 20220219 (but this might not be the case apparently).

max-rousseau commented 2 years ago

Last time I noticed it after power loss / reboot but I do believe I had done touchlink as well previously. The second time the issue was similar: use touchlink, then restart. Perhaps the first time I just noticed later due to restart happening later.

I have manually repaired everything now so I'm a bit hesitant to blow it up :)

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

Koenkk / zigbee2mqtt