search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
73.7k stars 30.82k forks source link

Zigbee ZHA async_initialize: all attempts have failed #89172

Closed matthewhadley closed 6 months ago

matthewhadley commented 1 year ago

The problem

Recently my zigbee network has become unreliable. Sensors will become unavailable and there are a number of errors in the logs.

This happened recently, after switching to SKyConnect: https://github.com/home-assistant/core/issues/87813#issue-1578791007

I thought it was resolved, but the issue came back. I have been trying to follow the guidelines at https://community.home-assistant.io/t/guide-for-zigbee-interference-avoidance-and-network-range-coverage-optimization/515752 - the skyconnect is attached to a long usb 2.0 extension.

I am wondering if there's a way to completely restart my zigbee network from scratch - would that be just a matter of removing all current devices?

Here are the errors in the log:

Logger: homeassistant.components.zha.core.channels.base
Source: components/zha/core/channels/base.py:486 
Integration: Zigbee Home Automation (documentation, issues) 
First occurred: 4:15:10 PM (30 occurrences) 
Last logged: 4:15:18 PM

[0x360E:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0x97B1:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0xD782:1:0x0406]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0x6697:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0x6697:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
Logger: zigpy.application
Source: runner.py:179 
First occurred: 4:15:27 PM (4 occurrences) 
Last logged: 4:20:08 PM

Unknown device AddrModeAddress(addr_mode=<AddrMode.NWK: 2>, address=0x8394)
Received relays from an unknown device: 0xb6ff

What version of Home Assistant Core has the issue?

core-2023.3.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

ZHA

Link to integration documentation on our website

https://www.home-assistant.io/integrations/zha/

Diagnostics information

config_entry-zha-0dedda09ac60fefd7e29f0f87d9f0cd9.json.txt

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 year ago

Hey there @dmulcahey, @adminiuga, @puddly, mind taking a look at this issue as it has been labeled with an integration (zha) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `zha` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign zha` Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)

zha documentation zha source (message by IssueLinks)

matthewhadley commented 1 year ago

I've decided to try and rebuild my zigbee network from scratch, moving to channel 25 which looks maybe less congested after viewing 2.4GHz wifi .

After adding several devices I had no errors, but at 20 devices I now see this error again:

Logger: homeassistant.components.zha.core.channels.base
Source: components/zha/core/channels/base.py:486 
Integration: Zigbee Home Automation (documentation, issues) 
First occurred: 10:00:21 PM (4 occurrences) 
Last logged: 10:00:26 PM

[0x3C9D:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0x3C9D:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0xBF22:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0xBF22:1:0x0006]: async_initialize: all attempts have failed: [TimeoutError(), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]

On a positive note, devices seem to be working and not showing as unavailable.

Similar to before, this error comes up when Home Assistant has restarted. Is there any way to use the information in the error to identify the devices involved?

Adminiuga commented 1 year ago

In square brackets, the 1st hex number is the nwk address of the device

matthewhadley commented 1 year ago

In square brackets, the 1st hex number is the nwk address of the device

Thanks @Adminiuga, do you know if there is a way to view detailed device information (including nwk address) for all devices? I see this information on each separate device page if I click Zigbee Info, but I'm hoping there is a more accessible list.

matthewhadley commented 1 year ago

I've found I can add nwk info to the zha-network-card https://github.com/dmulcahey/zha-network-card :)

Antexa commented 1 year ago

I got the same issue with the HA Yellow board

Logger: homeassistant.components.zha.core.channels.base
Source: components/zha/core/channels/base.py:486 
Integration: Zigbee Home Automation (documentation, issues) 
First occurred: 14:07:19 (69 occurrences) 
Last logged: 14:23:46

[0x53E2:11:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0x53E2:11:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0x7711:11:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0x7711:11:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0x7711:11:0x0008]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]

It was working pretty well for 1 month and this issue just appeared. Half of my Zigbee devices are now unreachable and some of them are very slow to respond (like unresponsive at all).

If you have an idea to locate the root cause

Dutchstyler commented 1 year ago

Looks the same as me, but was on older OS: https://github.com/home-assistant/core/issues/88806 Still have the same problem and can`t solve it. Have it only with the EcoDim dimmer.

matthewhadley commented 1 year ago

Since moving to channel 25 (away from 15) and restarting my network from scratch, things have improved. I have lots of zigbee router devices in my network, but it's difficult to interpret the "health" of the network in the visualization tab or to get insight into why the async_initialize: all attempts have failed warnings happen (they still happen and they're for mains powered devices). It's unclear if these are warnings that need action to be taken - they only seem to come up when home assistant restarts and the devices otherwise operate normally as best I can tell.

Dutchstyler commented 1 year ago

My problem seems to be solved by some config edit. In the ZHA config I have deselected "Turn on Advanced intensity during light transition" also I had tho change default transition from 10 to 1 second.

issue-triage-workflows[bot] commented 1 year ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

unimatrixx0 commented 1 year ago

i have the same problem. I switched from Sonoff ZB Bridge zu Skyconnect 30 devices are connected an work. Now ich want to connect 2 nous zigbee a1z

The new device are visible and can be conneced. But the connection is not stable (online/offline every 10 sec.)

Logfile: [0xD7B6:1:0x0b04]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')

System: RaspPi 4 Home Assistant 2023.6.2 Supervisor 2023.06.2 Operating System 10.2 Frontend 20230608.0 - latest

jgandhi19 commented 1 year ago

I am having a similar issue as well. I was using a Nortek Zigbee/Zwave stick and noticed since 6/10/23 many zigbee devices not responding to automations or manual toggling. Sometimes it would respond to on/off on a 5-10 second delay, sometimes the devices wouldn't respond to any toggle.

System: HA VM running on Unraid (qcow2 file) Home Assistant 2023.6.2 Supervisor 2023.06.2 Operating System 10.2 Frontend 20230608.0 - latest

To eliminate my build/config, or physical hardware, I purchased a Sonoff 3.0 Zigbee device and built a new HA VM from scratch. The only configuration I made on the new build was add the ZHA integration and configure with the Sonoff 3.0 (Nortek is unplugged). I scanned and paired a few zigbee devices and am experiencing the same issue. This is a completely new build, all up to date.

System: HA VM running on Unraid (NEW qcow2 file) Home Assistant 2023.6.2 Supervisor 2023.06.2 Operating System 10.3 Frontend 20230608.0 - latest

I have attached my log file with ZHA debugging turned on. Please let me know if there is anything I can help provide for further diagnosis.

home-assistant_zha_2023-06-16T16-39-15.907Z.log

mkanet commented 1 year ago

Recently, after each reboot, I get the below warnings in the HA log file. However, all my Zigbee devices appear to be working okay. Did something change in the Zigbee integration to cause this? I haven't changed anything on my zigbee network... these warnings just started to appear one day after I upgraded Home Assistant... I just can't remember which version caused this.

2023-06-21 17:38:57.401 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0xE843:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
2023-06-21 17:38:57.723 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x31DD:1:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
2023-06-21 17:38:57.759 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0xE843:1:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
2023-06-21 17:38:57.968 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x31DD:1:0x0008]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
2023-06-21 17:38:58.066 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0xEDD2:1:0x0406]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
2023-06-21 17:38:58.122 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0xE843:1:0x0008]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
2023-06-21 17:38:58.294 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x8C08:1:0x0002]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
2023-06-21 17:38:58.764 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x31DD:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
2023-06-21 17:38:59.040 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0xEDD2:1:0x0002]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
2023-06-21 17:38:59.150 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x8C08:1:0x0406]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
unimatrixx0 commented 1 year ago

now I have changed from Skyconnect-Stick to SONOFF Zigbee 3.0 USB Dongle Plus. ==> Problem solved - no errors, no problem with new devices, no disconnects since 1 Week :-)

thitcher commented 1 year ago

Still the same problem here with sky connect, the devices worked well but now the mesh begins to get unreliable and slow. All attempts to fix this have failed.

`Logger: homeassistant.components.zha.core.cluster_handlers Source: components/zha/core/cluster_handlers/init.py:480 Integration: Zigbee Home Automation (documentation, issues) First occurred: 06:21:42 (56 occurrences) Last logged: 06:22:12

[0x49F8:1:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')] [0x29A1:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')] [0x29A1:1:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')] [0xF475:1:0x0001]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), TimeoutError()] [0xF475:1:0x0400]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), TimeoutError()]`

Dutchstyler commented 1 year ago

Still the same problem here with sky connect, the devices worked well but now the mesh begins to get unreliable and slow. All attempts to fix this have failed.

`Logger: homeassistant.components.zha.core.cluster_handlers Source: components/zha/core/cluster_handlers/init.py:480 Integration: Zigbee Home Automation (documentation, issues) First occurred: 06:21:42 (56 occurrences) Last logged: 06:22:12

[0x49F8:1:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')] [0x29A1:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')] [0x29A1:1:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')] [0xF475:1:0x0001]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), TimeoutError()] [0xF475:1:0x0400]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), TimeoutError()]`

My experience is that is takes a few days before the mash network has arranged itself. Just wait a day and you should see a change in LQI and RSSI. Sometimes better or worse.

thitcher commented 1 year ago

The network was reliable for 2 months. Now it began to get unreliable. I switched off the coordinator for 30minutes to force the clients to get into panic mode, then I reconnected it and wait 2-3 days. Nothing changed. Then I tried to re-configure the coordinator (disabled "Turn on Advanced intensity during light transition" and changed the light transition time from 5 to 1 second). After that, the network was reliable :-) but only for 48h :-( So, now I bought a Sonoff Stick (p-version) and will try it with that.

thitcher commented 1 year ago

OK, today, the Sonoff USB dongle arrives. I flashed the latest FW and put it in. Then I migrated the existing radio from the sky connect stick to the Sonoff, do a restart and... Tadaa, everything works like a charm! No more errors, not during the startup process and not after starting my automations. So at the moment I think it is still a problem with the sky connect stick. Will wait and observe this a few days to get sure.

jinovak commented 1 year ago

I have the same experience. I used Raspberry Pi 4 and Raspbee II, and I never had any issues. Then I switched (changed the radio) to Skyconnect, and since then, there is no day with reliable network (almost 3 months of frustration). In the log, I constantly see "all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')" I also bought Sonoff dongle and flashed the repeater firmware, but nothing improved (except much better LQI). I have around 50 Zigbee devices, half of which are routers. It seems to me that Skyconnect is also a problem. @thitcher, please give me an update after 14 days if everything is still stable. I would try it too; right now, irrigation, blinds, and other things occasionally don't work, and it's hell.

thitcher commented 1 year ago

I have the same experience. I used Raspberry Pi 4 and Raspbee II, and I never had any issues. Then I switched (changed the radio) to Skyconnect, and since then, there is no day with reliable network (almost 3 months of frustration). In the log, I constantly see "all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')" I also bought Sonoff dongle and flashed the repeater firmware, but nothing improved (except much better LQI). I have around 50 Zigbee devices, half of which are routers. It seems to me that Skyconnect is also a problem. @thitcher, please give me an update after 14 days if everything is still stable. I would try it too; right now, irrigation, blinds, and other things occasionally don't work, and it's hell.

Hi, until now, the Sonoff works quite well, no such errors anymore. But LQI is not as good as with the sky connect it shows some weird values (e.g. lqi 3 for some devices) but connection is stable and works fine. Maybe this occurs because I restarted the HA a lot of times and it needs some days to settle...

Sometimes I get errors from my Aqara illumination sensor, but it works with no problems. No delivery failures anymore since the Sonoff does the job

Unbenannt Unbenannt

borodaaaaa commented 1 year ago

Have the same with con bee2 stick, at certain point all zig bee device start to be super laggy, full restart of HA make it better for some time, but i keep seeing these messages in the logs after every reboot

malfroid commented 1 year ago

After years of stable use, today I switched from a Conbee2 stick to SkyConnect. Everything seemed to be working fine for a few hours, but now everything started failing with similar issues as described above. Reboot does not fix it.

thitcher commented 1 year ago

@jinovak 14 days already over now. No problems with sonoff stick at all. Everything works fine. So it seems that the sky connect stick is the problem.

gadinbot commented 1 year ago

I have the con bee2, has been solid for the last 2 plus years, not all of a sudden, the network very unreliable. i have just ordered the sonoff e stick. fingers crossed.

update: Sonoff Zigbee 3.0 e arrived and once migrated everything came back up with he exception of one led controller. I tried setting it to pair but is will connect and then go back into pairing mode

mr2web commented 1 year ago

Homeassistant version: Core 2023.08 ZigBee adapter: ConBee2 (running the latest firmware) ZigBee integration: ZHA

I too have during the last good 9 month-ish experienced the same issue as all the testimonies above. Am using a ConBee2 stick for a few years without any issues then after an HA update suddenly lagg and instability became a common experience on a pretty much daily basis. Unfortunately I do not recall what was the last stable working HA revision but if I remember correctly I have posted in regard to this in the past without any solution found.

An alarming issue I do experience is that HA seem to loose the connection to the dongle and after a restart of HA it finds the dongle just fine for some time until it drops the connection again...

Else I do get the following error message keep appearing in my logs. This might not have anything to do with the drop to the dongle though.

[0xFDE9:1:0x0008]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>'), DeliveryError('Failed to deliver packet: <TXStatus.NWK_ROUTE_DISCOVERY_FAILED: 208>')]

I have reset all my ZigBee devices and created a new network and so on. All good for a few days then the error have come back and HA keep dropping its connection to the dongle.

This is very frustrating as I have automations that cares for my Hydroponic farm and when I experiences stability issues the farm can take great damage. Going away on vacation used to be a breeze, now its a constant worry...

I hope someone might crack the issue and come up with a solution. Please do note that I'm very happy with HA and its community and these stability issues are "speedbumps" but yet quite annoying.

Keep up the good work everyone! :-)

egelinotte commented 1 year ago

Guys, I would like to get a solid answer on whether or not the issue comes from the SkyConnect. This problem is painful and many automation frequently fail for me because my lights are driver by a Sonoff MiniL2. I am ready to switch to the Sonoff Dongle P if that really fix the problem.

puddly commented 1 year ago

Try out 2023.8.2, there was a bugfix in this release that may improve connectivity in poor environmental conditions.

egelinotte commented 1 year ago

Try out 2023.8.2, there was a bugfix in this release that may improve connectivity in poor environmental conditions.

I have it already and it did not help.

lux4rd0 commented 1 year ago

Same here. .2 didn't solve my issues. Although it does allow me to disable and re-enable automations that are stuck without freezing up the UI.

danpeig commented 1 year ago

Hi, I had the same problems you are reporting here with a Sonoff Zigbee Dongle E (chip EFR32MG21). Some devices would simply fail respond with the DELIVERY_FAILED error and after some trials they would magically wake.

I tried all firmware versions for this Dongle, from 6 to 7.3. The issue never went away.

Then I went back to my trusty Sonoff Zigbee Dongle P (chip CC2652) with KoenKK firmware. Everything went perfectly well. No device lags or failures. Note that the network configuration never changed (channel, IEEE, keys, etc...)

Today I received my Skyconnect (chip EFR32MG21) and immediately upgraded to the latest official firmware (7.2). The excitement of the new device went away as soon as I started to have the same problems I had with the Dongle E.

Back to Dongle P and my network is perfect!

Conclusion: The problem is with the EFR32MG21 based devices. I feel that this chipset/firmware is very "impatient" and does not wait enough for the responses from mesh.

Grey-Sheep commented 1 year ago

Same issue here. I have updated the Skyconnect firmware to the latest offered, and still no joy. Not looking good should the day ever come that CM4s are available and I try and migrate to my Home Assistant Yellow.......

mkanet commented 1 year ago

@Grey-Sheep just an FYI, this doesn't seem to be related to Skyconnect. I have this same problem with a completely different ZHA dongle: HUSBZB-1 (with ZHA and Zwave radios). The common denominator is Home Assistant. Also, I don't use Home Assistant Yellow. I use Home Assistant VMWare VM. Whatever this issue is, I didn't have this issue 4 months ago.

mkanet commented 1 year ago

@danpeig @matthewhadley don't think the issue is specific to EFR32MG21. I use the popular dual radio HUSBZB-1 USB dongle with a Silicon Labs EM3581 chip. This problem was introduced around the same time this Github issue was opened. It seems like the common denominator is the HA Zigbee Integration for Home Assistant.

matthewhadley commented 1 year ago

Since changing my zigbee channel https://github.com/home-assistant/core/issues/89172#issuecomment-1458518932 my network has been stable.

This is probably a separate matter, but I will mention here that I have recently been having issues with ZHA not being able to initialize on Homeassistant (re)start, with these kind of errors:

Logger: homeassistant.components.zha.core.cluster_handlers
Source: components/zha/core/cluster_handlers/__init__.py:537 
Integration: Zigbee Home Automation (documentation, issues) 
First occurred: 12:26:42 PM (56 occurrences) 
Last logged: 12:26:44 PM

[0x7665:1:0x0702]: async_initialize: all attempts have failed: [ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running')]
[0xF243:1:0x0006]: async_initialize: all attempts have failed: [ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running')]
[0xFFA2:1:0x0406]: async_initialize: all attempts have failed: [ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running')]
[0xC88B:1:0x0702]: async_initialize: all attempts have failed: [ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running')]
[0x500E:1:0x0702]: async_initialize: all attempts have failed: [ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running'), ControllerError('ApplicationController is not running')]

Logger: zigpy.application
Source: /usr/local/lib/python3.11/site-packages/zigpy/application.py:200 
First occurred: 12:26:23 PM (1 occurrences) 
Last logged: 12:26:23 PM

Logger: homeassistant.config_entries
Source: config_entries.py:429 
First occurred: 12:26:23 PM (1 occurrences) 
Last logged: 12:26:23 PM

Config entry 'Zigbee Home Automation' for zha integration not ready yet; Retrying in background

Couldn't start application
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 197, in startup
    await self.connect()
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 141, in connect
    await ezsp.startup_reset()
  File "/usr/local/lib/python3.11/site-packages/bellows/ezsp/__init__.py", line 151, in startup_reset
    await self._gw.wait_for_startup_reset()
  File "/usr/local/lib/python3.11/site-packages/bellows/uart.py", line 175, in wait_for_startup_reset
    await self._startup_reset_future
ConnectionResetError: Remote server closed connection

Logger: bellows.uart
Source: runner.py:186 
First occurred: 12:26:23 PM (1 occurrences) 
Last logged: 12:26:23 PM

Lost serial connection: ConnectionResetError('Remote server closed connection')

When this happens the network is totally unresponsive (it didn't start) and I have to reload the integration to fix. It seems similar to https://github.com/home-assistant/core/issues/99750 and I'm hoping it's resolved by https://github.com/home-assistant/core/pull/99764

sebastian3107 commented 1 year ago

After years of stable use, today I switched from a Conbee2 stick to SkyConnect. Everything seemed to be working fine for a few hours, but now everything started failing with similar issues as described above. Reboot does not fix it.

I have made the same switch and can report the same issue. One would assume that SkyConnect should be the dongle that has the best support for Home Assistant, but after migrating to SkyConnect I started seeing the 102 errors (delivery failed). What is strange is, that some smart plugs still report the sensor values like current and voltage just fine, I am just unable to turn them on / off.

thitcher commented 1 year ago

I can say that there have been no such problems with the Sonoff Dongle P for two months. I sometimes have error messages, but these are problems that I created myself. But I don't have any delivery failed messages anymore.

matthewhadley commented 1 year ago

Upgrading to 2023.9.2 did not fix the error preventing ZHA from starting :(

I was on the multipan protocol for SkyConnect to support both zigbee and thread. I've decided to go back to using it exclusively for Zigbee. Since doing that Zigbee starts ok on homeassistant start, but the failed to deliver message errors are back...

Logger: homeassistant.components.zha.core.cluster_handlers
Source: components/zha/core/cluster_handlers/__init__.py:537 
Integration: Zigbee Home Automation (documentation, issues) 
First occurred: 11:07:58 AM (32 occurrences) 
Last logged: 11:08:05 AM

[0x295F:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0x295F:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0xD444:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0x2715:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]
[0xEC05:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]

I cannot tell if it's related, but the supervisor logs also show these errors during startup:


23-09-13 11:05:15 WARNING (MainThread) [supervisor.misc.tasks] Watchdog miss API response from Home Assistant
23-09-13 11:07:14 INFO (MainThread) [supervisor.homeassistant.core] Wait until Home Assistant is ready
23-09-13 11:07:15 ERROR (MainThread) [supervisor.misc.tasks] Watchdog found a problem with Home Assistant API!
23-09-13 11:07:15 ERROR (MainThread) [supervisor.misc.tasks] Home Assistant watchdog reanimation failed!```
dexxa9879 commented 1 year ago

Exactly the same problem: the network becomes slow and devices disappear from the network.

Jozo81 commented 1 year ago

Same problem here. Moved to ZHA few months ago when started to use skyconnect (RaspPi4). At first everything worked perfectly but this week started to get EmberStatus.DELIVERY_FAILED: 102. Reloading Zha "fixes" the problem for short periods but then same errors come back again.

MikeDeltaHH commented 1 year ago

I have HA installed on an Intel NUC, use ZHA via SkyConnect and since a while exactly the same problem - the SkyConnect is connected to a USB 2.0 port with a 1m extension cable, any USB 3.0 interference sources are not in range. Every few days the same error pattern - <EmberStatus.DELIVERY_FAILED: 102> Some of my Zigbee devices are suddenly no longer accessible or automations fail with the above message - if I restart HA then everything works again for a few days…

danpeig commented 1 year ago

I can say that there have been no such problems with the Sonoff Dongle P for two months. I sometimes have error messages, but these are problems that I created myself. But I don't have any delivery failed messages anymore.

ZigBee Dongle P is rock stable!

Jozo81 commented 1 year ago

Tested with Skyconnect and Sonoff zdongle-E: Same errors with both, <EmberStatus.DELIVERY_FAILED: 102> etc. Then migrated radio to old Sonoff dongle (Texas_Instruments_TI_CC2531) and everything works problem free.

alexandoe commented 1 year ago

I had the same issye with async_initalize issues for some devices in the network. Rebooting and shutting down did not solve the issue for me. I disabled the support for Multiprotocol (Thread) in my SkyConnect since no devices use Thread in my installation and immediatly all devices started communicating again.

stratus-ss commented 1 year ago

this problem just started to plague me today. I haven't been able to get any deterministic causes. I do not have SkyConnect. I am currently in the process of updating from 2023.10.3 to 10.5, crossing my fingers that this will work.

I am using a Sonoff 3.0 dongle. I even went to the step of moving my 2.4ghz wifi to channel 11 just in case that was the problem

Grey-Sheep commented 1 year ago

In trying to track it down, I started with a clean install of HA, and have been slowly moving across the Zigbee devices one at a time. I'm using the SkyConnect Dongle. All was fine until I went over 50 devices, and now it is just awful. Devices dropping off the network, not responding, responding after a few seconds and generally being an absolute pain. It looks so much like interference, but why start now? I will now reduce the devices to below 50 and see if the issue goes away....

stratus-ss commented 1 year ago

I only have 36 devices which is the sad thing. I have several routers (wired outlets for zigbee)

Edit: I got a 1 meter USB extension cable and moved the Sonoff 3 USB dongle and that hasn't made any difference

Edit 2: I moved my Zigbee channel to 11... based on this website that should mean I am not overlapping

image

Still having the same problem. I have dropped 5 zigbee devices from my network and still cannot communicate with most of them

thitcher commented 1 year ago

I have 107 devices now , no problems with Sonoff dongle P, channel 25

kylepyke commented 11 months ago

I'm using a ZigStar UZG-01 LAN Zigbee coordinator, which uses the same chip as the Sonoff-P dongle (CC2652P).

I'm having the following error, constantly:

Logger: zigpy.application
Source: runner.py:188
First occurred: 2:48:29 AM (13 occurrences)
Last logged: 2:49:04 AM

    Received relays from an unknown device: 0xFF93
    Unknown device AddrModeAddress(addr_mode=<AddrMode.NWK: 2>, address=0xFF93)
    Unknown device AddrModeAddress(addr_mode=<AddrMode.NWK: 2>, address=0x9D70)
    Received relays from an unknown device: 0xA995
    Unknown device AddrModeAddress(addr_mode=<AddrMode.NWK: 2>, address=0xA995)
thitcher commented 11 months ago

I'm using a ZigStar UZG-01 LAN Zigbee coordinator, which uses the same chip as the Sonoff-P dongle (CC2652P).

I'm having the following error, constantly:

Logger: zigpy.application
Source: runner.py:188
First occurred: 2:48:29 AM (13 occurrences)
Last logged: 2:49:04 AM

    Received relays from an unknown device: 0xFF93
    Unknown device AddrModeAddress(addr_mode=<AddrMode.NWK: 2>, address=0xFF93)
    Unknown device AddrModeAddress(addr_mode=<AddrMode.NWK: 2>, address=0x9D70)
    Received relays from an unknown device: 0xA995
    Unknown device AddrModeAddress(addr_mode=<AddrMode.NWK: 2>, address=0xA995)

Had the same problem, in my case it was a zigbee device that wasn't de-registered properly

JohnWBour commented 11 months ago

Same erratic issue here with SkyConnect (was sometimes the case before with POP) and please note that I moved all wifi chan to the end and change Zigbee Channel from 25 to 20. The fact is Zigbee integration seems to be overloaded and could not reach the light switchs or get Zlinky data. The workaround is a zigbee integration restart but ofter directly HA restart and everything get back for a while

I'll also add that I completely reinstalled a fresh HA on HAOS only 3 weeks ago and I also get a 2m wire to isolate the Skyconnect key from USB or anything else...