Open zsmb13 opened 8 months ago
Hey there @dmulcahey, @adminiuga, @puddly, @thejulianjes, mind taking a look at this issue as it has been labeled with an integration (zha
) you are listed as a code owner for? Thanks!
(message by CodeOwnersMention)
zha documentation zha source (message by IssueLinks)
Got one more example, though I don't expect there's much new info in here:
2023-12-21 17:39:59.603 WARNING (MainThread) [bellows.zigbee.application] Watchdog heartbeat timeout: TimeoutError()
2023-12-21 17:39:59.603 WARNING (MainThread) [zigpy.application] Watchdog failure
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/bellows/ezsp/protocol.py", line 68, in command
return await future
^^^^^^^^^^^^
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 661, in _watchdog_loop
await self._watchdog_feed()
File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 893, in _watchdog_feed
(res,) = await self._ezsp.readCounters()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/bellows/ezsp/protocol.py", line 67, in command
async with asyncio_timeout(EZSP_CMD_TIMEOUT):
File "/usr/local/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
raise TimeoutError from exc_val
TimeoutError
2023-12-21 17:39:59.635 DEBUG (MainThread) [zigpy.application] Connection to the radio has been lost: TimeoutError()
2023-12-21 17:39:59.638 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] Connection to the radio was lost: TimeoutError()
2023-12-21 17:39:59.638 DEBUG (MainThread) [zigpy.application] Stopping watchdog loop
2023-12-21 17:39:59.638 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] Shutting down ZHA ControllerApplication
...
2023-12-21 17:40:08.901 DEBUG (MainThread) [homeassistant.components.zha] Failed to set up ZHA
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/bellows/uart.py", line 261, in reset
return await self._reset_future
^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/__init__.py", line 163, in async_setup_entry
zha_gateway = await ZHAGateway.async_from_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 182, in async_from_config
await instance.async_initialize()
File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 200, in async_initialize
await app.startup(auto_form=True)
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 223, in startup
await self.connect()
File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 139, in connect
await ezsp.startup_reset()
File "/usr/local/lib/python3.11/site-packages/bellows/ezsp/__init__.py", line 119, in startup_reset
await self.reset()
File "/usr/local/lib/python3.11/site-packages/bellows/ezsp/__init__.py", line 145, in reset
await self._gw.reset()
File "/usr/local/lib/python3.11/site-packages/bellows/uart.py", line 260, in reset
async with asyncio_timeout(RESET_TIMEOUT):
File "/usr/local/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
raise TimeoutError from exc_val
TimeoutError
Seems like I found a solution that fixes this without unplugging physically, though this is still far from great:
sudo sh -c "echo 0 > /sys/bus/usb/devices/usb1/authorized"
sudo sh -c "echo 1 > /sys/bus/usb/devices/usb1/authorized"
Not sure if it's possible to skip the Home Assistant restart step. Perhaps it's enough to wait for a while between unmount/remount, I'll try it next time it fails.
Could you post a complete ZHA debug log from startup up until the crash?
Is there a good way to redact these logs, or share them privately at least? There's a lot of data in there, and one of the logs I have is also quite large (500 MBs).
Thanks! Compressing the log usually shrinks it significantly. If you want to send it privately, you can email it/a link to it to puddly3@gmail.com
.
Sent the logs over email. Thank you!
Had it fail again just this morning. I've sent more logs over email.
This time I tried unmounting and remounting the USB device from the command line without restarting Home Assistant. This was unsuccessful, ZHA initialization got stuck for at least 5 minutes, at which point I restarted Home Assistant. That restart after unmounting/remounting fixed things, and ZHA initialized successfully.
So it seems like the unmount/remount steps and a HA restart are both required for a fix, but the ordering of these can vary a bit.
Unfortunately, it looks like your dongle is just outright crashing. Could your hypervisor/host OS be preventing it from communicating?
Did this work in the past? Can you test an older version of ZHA? Your dongle is crashing during a quiet period so it just happens with no clear cause.
One thing to try is to downgrade to the last stable release of EZSP 6.9.x.
I'll see if I can find any problems around how the device is mounted into Docker, but I have nothing that would indicate that the problem is there. I've set up things recently, so I don't really have experience with previous versions of things. I'll also see if I can perform a firmware downgrade and if that helps.
Is there any chance that this could be a hardware fault, could it be worth returning the dongle and replacing it with a new one?
Hello, I have the same problems. For Zigbee, nothing works correctly since version 2023.12.0. I carried out tests with zigbee2mqtt and everything works perfectly, the problem is ZHA. I use a Sonoff zigbee 3.0 version P dongle. It is no. longer even possible to detect new hardware
My Zigbee also died after the upgrade and as it's in a vm I restored to the snapshot to before the upgrade and everything worked fine again. The hardware is working normally on: Core 2023.11.2 Supervisor 2023.12.0 Operating System 11.1 Frontend 20231030.2 However when I upgrade Core to 2023.12.4 I loose Zigbee totally. I have tried this several times and each time Zigbee dies I roll it back to the before upgrade snapshot and it's fine again so I know it's not the hardware. I have not pulled logs yet as I'm not sure which ones are viable for troubleshooting this issue currently. Hardware is Sonoff 3.0 USB Dongle Plus , Model ZBDongle-E
Hello, For me it doesn't work properly when restoring an earlier version like 2023.12.2. Big problems persist.
@sidwin9 You aren't using a Sonoff Dongle-E. Please don't cross-post to multiple issues.
Hi, I have the same problem core-2023.12.4 supervisor-2023.12.0 SONOFF Zigbee 3.0 USB Dongle Plus (ZBDongle-E)
Got a bit more on the issue, it's still happening, and what's worse is that it can seemingly take down Home Assistant entirely. After ZHA started failing today, I restarted Home Assistant (which on its own did not fix the issue), and after 15 minutes it stopped responding altogether. It's one thing that ZHA is failing, but it should not be able to make Home Assistant unaccessible.
The Docker container is still running and consuming resources:
However, it stopped producing logs at 21:13 with the very last message in the log being this:
2023-12-30 21:13:09.330 DEBUG (MainThread) [bellows.ezsp] Resetting EZSP
Hi, I got it working today. I Believe it might have been Silicon Labs Multiprotocol (Current version: 2.3.2) causing the issue but I was playing around with Silicon Labs Flasher Add-on. Could be relevant.
Update: For me the latest updates seemed to have resolved whatever issue there was, Zigbee working normally now. Running: Core 2024.1.2 Supervisor 2023.12.0 Operating System 11.3 Frontend 20240104.0
@scality-jeff What device and firmware are you running?
@zsmb13 I've got SONOFF Zigbee 3.0 USB Dongle Plus V2 (from home assistant) From product package - Zigbee 3.0 USB Model ZBDongle-E This is in vmware with the usb dongle as pass through from the vm host to the vm.
Try the release later today. It contains stability improvements for SI radios.
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
I have a restart script for this now that remounts the dongle and restarts HA (this triggers once every couple days) and I also reboot HA every night, which sort of solves the problem, but the underlying issue of the dongle going unresponsive is still very much there.
I also faced this issue and I cannot find a way to solve it. In my case, it's about the latest HA running on HA Green. I have tried ZigBee Dongle E with the pretty old FW - 6.10.3.0 build 297 and also the more up-to-date one - 7.x. In both cases after max one week the Zigbee died... ;(
Still getting this, more and more recently. These are the exceptions I can find in my logs now:
2024-07-14 21:23:22.038 DEBUG (MainThread) [homeassistant.components.zha] Failed to set up ZHA
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/__init__.py", line 152, in async_setup_entry
zha_gateway = await ZHAGateway.async_from_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 197, in async_from_config
await instance.async_initialize()
File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 215, in async_initialize
await app.startup(auto_form=True)
File "/usr/local/lib/python3.12/site-packages/zigpy/application.py", line 233, in startup
await self.connect()
File "/usr/local/lib/python3.12/site-packages/bellows/zigbee/application.py", line 149, in connect
await ezsp.startup_reset()
File "/usr/local/lib/python3.12/site-packages/bellows/ezsp/__init__.py", line 125, in startup_reset
await self.reset()
File "/usr/local/lib/python3.12/site-packages/bellows/ezsp/__init__.py", line 151, in reset
await self._gw.reset()
TimeoutError
2024-07-14 21:23:22.041 INFO (MainThread) [universal_silabs_flasher.flasher] Probing ApplicationType.GECKO_BOOTLOADER at 115200 baud
2024-07-14 21:23:22.041 DEBUG (MainThread) [zigpy.serial] Opening a serial connection to '/dev/ttyACM0' (115200 baudrate)
2024-07-14 21:23:23.042 INFO (MainThread) [universal_silabs_flasher.flasher] Probing ApplicationType.CPC at 460800 baud
2024-07-14 21:23:23.043 DEBUG (MainThread) [zigpy.serial] Opening a serial connection to '/dev/ttyACM0' (460800 baudrate)
2024-07-14 21:23:24.044 INFO (MainThread) [universal_silabs_flasher.flasher] Probing ApplicationType.CPC at 115200 baud
2024-07-14 21:23:24.044 DEBUG (MainThread) [zigpy.serial] Opening a serial connection to '/dev/ttyACM0' (115200 baudrate)
2024-07-14 21:23:25.046 INFO (MainThread) [universal_silabs_flasher.flasher] Probing ApplicationType.CPC at 230400 baud
2024-07-14 21:23:25.046 DEBUG (MainThread) [zigpy.serial] Opening a serial connection to '/dev/ttyACM0' (230400 baudrate)
2024-07-14 21:23:26.047 INFO (MainThread) [universal_silabs_flasher.flasher] Probing ApplicationType.EZSP at 115200 baud
2024-07-14 21:23:26.048 DEBUG (MainThread) [zigpy.serial] Opening a serial connection to '/dev/ttyACM0' (115200 baudrate)
2024-07-14 21:23:58.055 DEBUG (MainThread) [homeassistant.components.zha.repairs.wrong_silabs_firmware] Failed to probe application type
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/repairs/wrong_silabs_firmware.py", line 87, in probe_silabs_firmware_type
await flasher.probe_app_type()
File "/usr/local/lib/python3.12/site-packages/universal_silabs_flasher/flasher.py", line 235, in probe_app_type
raise RuntimeError("Failed to probe running application type")
RuntimeError: Failed to probe running application type
I'm now on Core 2024.7.2, and things definitely got worse recently. My instance can't go 24 hours without the Zigbee connection failing as described above. Sometimes it falls over every hour.
I'm having exactly the same issue as @zsmb13 . My Sonoff Dongele E just disconnected from the USB and doesn't show up in hardware anmore. I need to unplug and re plug it in. It seems to happen overnight mainly when everything is quiet. It onlt started doing this a couple months ago after I upgraded my verions of HAOS.
I gave up and bought a Home Assistant Connect ZBT-1. No issues for 2.5 weeks now since I'm using it. Migration was a couple minutes, well worth it.
thanks @zsmb13 I think I'll try that also. What did you need to do to migrate? I have my HAOS installed on a mini-pc so I'm assumning I can ust plug into one of the USB ports?
I run in Docker with ZHA. I pretty much followed these steps https://connectzbt1.home-assistant.io/migrate-zigbee-zha-only/
Also checked a couple other places that described this, YouTube, forum, etc., but this guide seems to be accurate. Also worth doing just a regular backup before starting the migration, so that you have a file to restore from, whatever happens.
The problem
I have a setup with a couple dozen Zigbee devices, coordinated by a SONOFF ZigBee 3.0 Dongle E. Every couple days the dongle / ZHA crash in an unrecoverable way, which I can only solve by physically unplugging and replugging the USB stick. I'm using firmware 7.3.1.0 build 176, flashed by the web-based tool.
I've captured a couple stacktraces now when this happened, hopefully there's something meaningful in there. Let me know what other details I can provide.
What version of Home Assistant Core has the issue?
core-2023.12.3
What was the last working version of Home Assistant Core?
No response
What type of installation are you running?
Home Assistant Container
Integration causing the issue
zha
Link to integration documentation on our website
https://www.home-assistant.io/integrations/zha/
Diagnostics information
Diagnostics, captured after recovering from a failure already:
zha-4a8c1b710746a49bb72205002a49a0b7-Zigbee Coordinator-8733fd8a1920e10f5420c786d6c31417.json.txt
Example YAML snippet
No response
Anything in the logs that might be useful for us?
Additional information
No response