home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
72.09k stars 30.18k forks source link

Controller reporting "Jammed" and then "Ready", operations hang #104230

Open ember1205 opened 10 months ago

ember1205 commented 10 months ago

The problem

I am commonly seeing my USB device (Aeotec Z-Stick 7 Plus) report being "Jammed" and then shortly after report being "Ready." Around this time, any operations will fail to occur and the system does not go back and "recover" from missed actions / automations.

What version of Home Assistant Core has the issue?

System Information version | core-2023.11.2 -- | -- installation_type | Home Assistant OS dev | false hassio | true docker | true user | root virtualenv | false python_version | 3.11.6 os_name | Linux os_version | 6.1.59 arch | x86_64 timezone | America/New_York config_dir | /config
Home Assistant Cloud logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Home Assistant Supervisor host_os | Home Assistant OS 11.1 -- | -- update_channel | stable supervisor_version | supervisor-2023.11.3 agent_version | 1.6.0 docker_version | 24.0.6 disk_total | 228.5 GB disk_used | 6.1 GB healthy | true supported | true board | generic-x86-64 supervisor_api | ok version_api | ok installed_addons | Z-Wave JS (0.3.0), Let's Encrypt (4.12.9), Z-Wave JS UI (3.0.2)
Dashboards dashboards | 2 -- | -- resources | 0 views | 1 mode | storage
Recorder oldest_recorder_run | November 10, 2023 at 7:57 PM -- | -- current_recorder_run | November 16, 2023 at 3:32 PM estimated_db_size | 60.77 MiB database_engine | sqlite database_version | 3.41.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

zwave-js

Link to integration documentation on our website

No response

Diagnostics information

zwave_js-6d4b40be1e77c3e33d78bdf24d66a9c3-USB Controller-d288fb52eab027710676687815136717.json (1).txt

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

I can provide debug logs if useful for zwave-js: Please let me know if you want an entire day's worth of info or if you would prefer a certain amount of time leading up to and then after an event.

tony-park commented 8 months ago

The claim is that it's the 700/800 controllers, but my understanding is that this is happening with the 500 series as well. So, not sure that it is the same. Dunno.

They may not be developing anymore for 500 series, so may well never acknowledge that there was/is an issue with that series of controller.

russellmoran99 commented 8 months ago

@russellmoran99 in older versions, Z-Wave JS didn't react at all to a jammed controller, and instead would randomly mark devices as dead. If this behavior is preferred, you can revert to it by adding the following line to your Z-Wave addon configuration (switch to YAML mode):

disable_controller_recovery: true

New USB2.0 powered hub installed. Migrated to JS-UI and it worked for about 25 minutes…now it’s unresponsive again. I don’t want to wait 3-4 weeks from SiLabs to come up with a possible firmware. I truly don’t care if I need to buy a brand new system, how do I get this operable again?

markus99 commented 8 months ago

@russellmoran99 I threw in the towel 6 weeks ago, bought a Hubitat C8 and haven't looked back.

AlCalzone commented 8 months ago

@russellmoran99 Unfortunately there's nothing we can do if the hardware we're talking to is hanging or not behaving as it should.

@ember1205 the issue on 500 series is different from the ones on 700/800 series, but with roughly the same end result - the controller becomes unresponsive. In contrast to the 700/800 series issues, I don't have any hopes that this will get fixed though.

russellmoran99 commented 8 months ago

@russellmoran99 Unfortunately there's nothing we can do if the hardware we're talking to is hanging or not behaving as it should.

Is there new hardware I could use to get my system back up and running or are suggesting Zwave is done? I’m asking for a solution even if it’s new hardware. If there is not, it seems Zwave is done for now. Please advise.

ember1205 commented 8 months ago

@russellmoran99 Unfortunately there's nothing we can do if the hardware we're talking to is hanging or not behaving as it should.

Is there new hardware I could use to get my system back up and running or are suggesting Zwave is done? I’m asking for a solution even if it’s new hardware. If there is not, it seems Zwave is done for now. Please advise.

ZWave protocol is open source, chips are not. All hardware requires chipsets from SiliconLabs and they release the SDK's that firmware is built on. Recent SDK's indicate an issue in the 700/800 based platforms, but there is no information as to whether that is what we are seeing with these behaviors and errors in HA.

In other words, you can't swap hardware for a fix with HA.

AlCalzone commented 8 months ago

I’m asking for a solution

700/800 series should get fixed soon, I wouldn't expect a fix for the 500 series though.

Recent SDK's indicate an issue in the 700/800 based platforms, but there is no information as to whether that is what we are seeing with these behaviors and errors in HA.

It is. You'll have to take my word for it though.

tony-park commented 8 months ago

Hopefully the fix will be out in the next couple of weeks. If you can reduce your chatty devices to only report details you really need, the network stability is a lot better.

russellmoran99 commented 8 months ago

Hopefully the fix will be out in the next couple of weeks. If you can reduce your chatty devices to only report details you really need, the network stability is a lot better.

@tony-park I'm using JS-UI. How do I do this? I appreciate it. I have googled it and the articles tell me the same thing you are telling me, but they don't explain how to do it. Thanks

ember1205 commented 8 months ago

Hopefully the fix will be out in the next couple of weeks. If you can reduce your chatty devices to only report details you really need, the network stability is a lot better.

@tony-park I'm using JS-UI. How do I do this? I appreciate it. I have googled it and the articles tell me the same thing you are telling me, but they don't explain how to do it. Thanks

You need to configure each device and turn off or turn down the automatic reporting it's doing. Get rid of what you don't explicitly need, change to basic reports instead of full reports, turn it off completely if you need it.

russellmoran99 commented 8 months ago

Hopefully the fix will be out in the next couple of weeks. If you can reduce your chatty devices to only report details you really need, the network stability is a lot better.

@tony-park I'm using JS-UI. How do I do this? I appreciate it. I have googled it and the articles tell me the same thing you are telling me, but they don't explain how to do it. Thanks

You need to configure each device and turn off or turn down the automatic reporting it's doing. Get rid of what you don't explicitly need, change to basic reports instead of full reports, turn it off completely if you need it.

@ember1205 I appreciate that, but again, how? What's the logical steps to doing this? I can see the device in JS-UI but not sure where to go to do this.

AlCalzone commented 8 months ago

Open the "Configuration" panel, search for parameters that sound like they affect reporting and change them where it makes sense.

ember1205 commented 8 months ago

@ember1205 I appreciate that, but again, how? What's the logical steps to doing this? I can see the device in JS-UI but not sure where to go to do this.

IMHO, it isn't overly intuitive, and "it depends" on the specific device. Generally, you click the down arrow to the left of the device and then parse through the various sub-sections to see what items are "auto reporting" for things like humidity level, temperature, power consumption, voltage, etc. and adjust those settings. In many cases, I find it simpler to use the built-in ZWave device page for each device that I need to configure because all of the settings are on single page once you click on "Configure".

russellmoran99 commented 8 months ago

Question Configuration Panel

Am I close this the configuration panel? I do not seem to be on the same page. Sorry to be a pain, just want this awful experience to be over.

ember1205 commented 8 months ago

You now need to drop every one of those down arrows on the right to see what settings are available to you. Once you change the settings, be sure to press the little "send" airplane icon or disk icon to save.

AlCalzone commented 8 months ago

These settings are only ever going to be in the "Configuration" panel, right at the bottom of the screenshot.

russellmoran99 commented 8 months ago

@ember1205 @AlCalzone @tony-park Great to know! I have went thru EVERY device I have and this is what most plugins have in configuration. I don't see anything specific to reporting. Is there anything to optimize here?

Question Configuration Panel2

tony-park commented 8 months ago

My guilty device was an aeotec energy monitor. But I also changed a couple of multisensor to just report values I'm interested in. Much improved since then. Probably get the jammed status twice a day now, as opposed to 3-4 times an hour.

ember1205 commented 8 months ago

My guilty device was an aeotec energy monitor. But I also changed a couple of multisensor to just report values I'm interested in. Much improved since then. Probably get the jammed status twice a day now, as opposed to 3-4 times an hour.

I have 12-15 Aeotec controllable plugs. Being that I actually key off of the power draw on about half of them to automate certain things, I do need to be informed when power draw levels change. But, some were configured to auto-report "everything" (voltage, current, wattage, kWh use, etc.) and do it frequently. After disabling the various details I don't need at all, disabling much of the interval-based auto reporting, and setting change thresholds for the variables that I -do- need, overall traffic on my network dropped of significantly and helped allow me to get 12-36 hours of operational time in between jammed status messages now.

AlCalzone commented 8 months ago

@russellmoran99 which devices are those? Either they have no configuration, or we don't have a config file for them. You may need to look at the device manual and use that "custom configuration" field to configure them.

russellmoran99 commented 8 months ago

@russellmoran99 which devices are those? Either they have no configuration, or we don't have a config file for them. You may need to look at the device manual and use that "custom configuration" field to configure them.

These are leviton, ge, and lutron plug ins.

AlCalzone commented 8 months ago

Seems for a lot of those we don't have the configuration parameters in our files. You'll have to go the manual route.

russellmoran99 commented 8 months ago

what is the manual route?

russellmoran99 commented 8 months ago

@AlCalzone

I have a FB friend selling me an older SiLab 700 series controller with 7.11 firmware on it. If I use that and leave it or flash it to 7.18 would I be able to back up my current zwave network using JS-UI Advanced Actions --> Backup --> Export and then import that into my new zwave controller when I get it? I would also configure JS-UI to point to the new USB controller and set the S0 Legacy Key, S2 Access Control Key, S2 Authenticated Key, S2 Unauthenticated Key and Network Keys as they were in the current (broken) zwave controller. Could it be that simple or would I have to completely exclude all devices from current network and include them in the new one? Thanks for any help.

AlCalzone commented 8 months ago

The manual route means actually looking at the device manual, figuring out which configuration parameter you want to change and enter number/size/value in the "custom configuration" fields you have there.

Migrating to 7.18 can be done if your Z-Stick 5 is on firmware 1.2. Just backup, then switch controllers and restore. If it doesn't work, you can always go back to the old controller.

russellmoran99 commented 8 months ago

@AlCalzone @tony-park @ember1205

I'm not sure if I have "fixed" my problem, but it's been about 12 hours, and my system has been operable. I unplugged ALL of my zwave plugins last night thinking that could be an issue. I also found 1 wall switch that is a Jasco Dimmer that seemed to be malfunctioning. I rebooted and my zwave controller was still inoperable. I had previously switched over from SmartThings and had left my ST Gen2 Hub on (in another room) without any problem over the last 2 months. I decided to turn it off and reboot my home assist again. When it came back on, everything is working. I wanted to see if there is a remote possibility that someone else has an existing zwave that could be causing interference?

ember1205 commented 8 months ago

@russellmoran99 - Radio interference is always something to be looking for. At 833MHz, there isn't "as much" stuff within a house that can interfere as with stuff like WiFi, but it can happen.

Could another ZWave system be causing interference? Sure - especially if it's close enough to items in your HA network, and even moreso if it's a central item in the HA network that many other devices on the network communicate through.

Your thought about another controller is an interesting one. I have another "hub" in the house that has been on and running since the very beginning w/ HA. I was disconnecting devices and then connecting them into HA. That effort is done at this point and I'll shut the hub down (with NO other changes) to see what impact there may be on my overall environment.

markus99 commented 8 months ago

@russellmoran99 @ember1205 I too had an old hub, SmartThings v2, prior to changing to Aeotec Gen5 USB stick<->HA. I transitioned ALL my ZWave devices (~30 wall switches, 2 AEOTEC Sensors, 5 plug outlets) when I made the switch. Unplugged ST Hub months (year+?) before September 2023 issues started.

Agree that it's likely not helping the issue, but it's also / most likely not causing it either...

russellmoran99 commented 8 months ago

@russellmoran99 - Radio interference is always something to be looking for. At 833MHz, there isn't "as much" stuff within a house that can interfere as with stuff like WiFi, but it can happen.

Could another ZWave system be causing interference? Sure - especially if it's close enough to items in your HA network, and even moreso if it's a central item in the HA network that many other devices on the network communicate through.

Your thought about another controller is an interesting one. I have another "hub" in the house that has been on and running since the very beginning w/ HA. I was disconnecting devices and then connecting them into HA. That effort is done at this point and I'll shut the hub down (with NO other changes) to see what impact there may be on my overall environment.

@ember1205 I'd love to hear your results once it's shut down. I rebooted my HA as well, not sure if that will help?

russellmoran99 commented 8 months ago

@AlCalzone

Not sure if this will help, but I decided to remove one of my "failed" nodes (#47). It is pretty much a ghost node as it is no longer powered on at all. When I did so, my zwave stick become unresponsive again. I will power cycle my system and see if it comes back up and running. Here is the code:

2024-01-17 11:35:39.028 INFO APP: GET /assets/prismeditor.esm-274000ef.js 304 5.647 ms - - 2024-01-17 11:35:39.157 INFO APP: GET /health/zwave 301 1.008 ms - 191 2024-01-17 11:36:09.330 INFO APP: GET /health/zwave 301 0.935 ms - 191 2024-01-17 11:36:18.046 INFO APP: GET /api/store?path=/data/store/ee460c51.metadata.jsonl 200 129.203 ms - 3990467 2024-01-17 11:36:39.438 INFO APP: GET /health/zwave 301 1.204 ms - 191 2024-01-17 11:36:51.679 INFO Z-WAVE: Controller status: Controller is unable to transmit 2024-01-17 11:36:55.783 INFO Z-WAVE: Controller status: Controller is Ready 2024-01-17 11:36:55.788 ERROR Z-WAVE-SERVER: Failed to send the command after 5 attempts (ZW0202) ZWaveError: Failed to send the command after 5 attempts (ZW0202) at Driver.sendMessage (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5256:23) at Driver.sendCommandInternal (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5454:28) at Driver.sendSupervisedCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5499:27) at Driver.sendCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5547:30) at Proxy.set (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:138:24) at Proxy. (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:160:16) at ZWaveNode.setValue (/opt/node_modules/zwave-js/src/lib/node/Node.ts:1150:39) at NodeMessageHandler.handle (/opt/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:19:43) at Object.node (/opt/node_modules/@zwave-js/server/dist/lib/server.js:42:92) at Client.receiveMessage (/opt/node_modules/@zwave-js/server/dist/lib/server.js:125:99) 2024-01-17 11:36:55.815 INFO Z-WAVE: Controller status: Controller is unable to transmit 2024-01-17 11:36:59.925 INFO Z-WAVE: Controller status: Controller is Ready 2024-01-17 11:36:59.929 ERROR Z-WAVE-SERVER: Failed to send the command after 5 attempts (ZW0202) ZWaveError: Failed to send the command after 5 attempts (ZW0202) at Driver.sendMessage (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5256:23) at Driver.sendCommandInternal (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5454:28) at Driver.sendSupervisedCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5499:27) at Driver.sendCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5547:30) at Proxy.set (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:138:24) at Proxy. (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:160:16) at ZWaveNode.setValue (/opt/node_modules/zwave-js/src/lib/node/Node.ts:1150:39) at NodeMessageHandler.handle (/opt/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:19:43) at Object.node (/opt/node_modules/@zwave-js/server/dist/lib/server.js:42:92) at Client.receiveMessage (/opt/node_modules/@zwave-js/server/dist/lib/server.js:125:99) 2024-01-17 11:37:09.565 INFO APP: GET /health/zwave 301 1.093 ms - 191 2024-01-17 11:37:29.049 INFO APP: GET /api/store?path=/data/store/ee460c51.values.jsonl 200 15.805 ms - 410594 2024-01-17 11:37:39.695 INFO APP: GET /health/zwave 301 1.332 ms - 191 2024-01-17 11:38:09.809 INFO APP: GET /health/zwave 301 1.118 ms - 191 2024-01-17 11:38:24.342 INFO APP: GET /api/store?path=/data/store/nodes.json 304 3.828 ms - - 2024-01-17 11:38:39.943 INFO APP: GET /health/zwave 301 0.986 ms - 191 2024-01-17 11:38:40.875 INFO APP: GET /api/store?path=/data/store/settings.json 304 4.322 ms - - 2024-01-17 11:38:54.129 INFO APP: GET /api/store?path=/data/store/users.json 200 4.720 ms - 221 2024-01-17 11:39:10.077 INFO APP: GET /health/zwave 301 1.019 ms - 191 2024-01-17 11:39:27.778 INFO Z-WAVE: Controller status: Controller is unable to transmit 2024-01-17 11:39:31.887 INFO Z-WAVE: Controller status: Controller is Ready 2024-01-17 11:39:31.892 ERROR Z-WAVE-SERVER: Failed to send the command after 5 attempts (ZW0202) ZWaveError: Failed to send the command after 5 attempts (ZW0202) at Driver.sendMessage (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5256:23) at Driver.sendCommandInternal (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5454:28) at Driver.sendSupervisedCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5499:27) at Driver.sendCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5547:30) at Proxy.set (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:138:24) at Proxy. (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:160:16) at ZWaveNode.setValue (/opt/node_modules/zwave-js/src/lib/node/Node.ts:1150:39) at NodeMessageHandler.handle (/opt/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:19:43) at Object.node (/opt/node_modules/@zwave-js/server/dist/lib/server.js:42:92) at Client.receiveMessage (/opt/node_modules/@zwave-js/server/dist/lib/server.js:125:99) 2024-01-17 11:39:31.919 INFO Z-WAVE: Controller status: Controller is unable to transmit 2024-01-17 11:39:36.030 INFO Z-WAVE: Controller status: Controller is Ready 2024-01-17 11:39:36.033 ERROR Z-WAVE-SERVER: Failed to send the command after 5 attempts (ZW0202) ZWaveError: Failed to send the command after 5 attempts (ZW0202) at Driver.sendMessage (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5256:23) at Driver.sendCommandInternal (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5454:28) at Driver.sendSupervisedCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5499:27) at Driver.sendCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5547:30) at Proxy.set (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:138:24) at Proxy. (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:160:16) at ZWaveNode.setValue (/opt/node_modules/zwave-js/src/lib/node/Node.ts:1150:39) at NodeMessageHandler.handle (/opt/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:19:43) at Object.node (/opt/node_modules/@zwave-js/server/dist/lib/server.js:42:92) at Client.receiveMessage (/opt/node_modules/@zwave-js/server/dist/lib/server.js:125:99) 2024-01-17 11:39:40.188 INFO APP: GET /health/zwave 301 1.087 ms - 191 2024-01-17 11:39:42.877 DEBUG SOCKET: User disconnected from V2MDpZURtpwUhGGsAAAN: transport close 2024-01-17 11:40:10.303 INFO APP: GET /health/zwave 301 1.900 ms - 191 2024-01-17 11:40:40.422 INFO APP: GET /health/zwave 301 0.924 ms - 191

UPDATE:

A reboot of the whole system worked. "Restart Home Assistant" did NOT work. "Reboot system" got my item responsive again. I still have "ghost" nodes and looks like I will have them for a while as I don't want to screw up my system now.

Ghost Nodes

UPDATE 2:

I wanted to rebuild my zwave network as I have removed all the plugins (about 15 of them). I rebuilt it and now the controller is unresponsive again. I rebooted the system and it's still unresponsive. I had created a FULL backup this am while everything was working so I will restore that and pray it is operational again.

Known working Backup has been restored, my zwave controller is still unresponsive. It seems running a "Rebuild" was not a good idea. :(

Error Log After Network Rebuild and attempting to turn on/off a few office lights.

2024-01-17 14:20:29.097 INFO Z-WAVE: [Node 043] Value updated: 38-0-currentValue 99 => 99 2024-01-17 14:20:29.105 INFO Z-WAVE: [Node 043] Value updated: 38-0-currentValue 99 => 99 2024-01-17 14:20:29.190 INFO Z-WAVE: [Node 043] Value updated: 38-0-currentValue 99 => 99 2024-01-17 14:20:29.493 INFO Z-WAVE: [Node 043] Value updated: 38-0-currentValue 99 => 99 2024-01-17 14:20:29.788 INFO Z-WAVE: [Node 043] Value updated: 38-0-currentValue 99 => 99 2024-01-17 14:20:30.133 INFO Z-WAVE: [Node 043] Value updated: 38-0-currentValue 99 => 99 2024-01-17 14:20:30.152 INFO Z-WAVE: [Node 043] Value updated: 38-0-currentValue 99 => 99 2024-01-17 14:20:30.516 INFO Z-WAVE: [Node 043] Value updated: 38-0-currentValue 99 => 99 2024-01-17 14:20:30.847 INFO Z-WAVE: [Node 043] Value updated: 38-0-currentValue 99 => 99 2024-01-17 14:20:43.441 INFO APP: GET /health/zwave 301 1.204 ms - 191 2024-01-17 14:21:13.569 INFO APP: GET /health/zwave 301 1.388 ms - 191 2024-01-17 14:21:43.691 INFO APP: GET /health/zwave 301 1.253 ms - 191 2024-01-17 14:22:13.828 INFO APP: GET /health/zwave 301 1.131 ms - 191 2024-01-17 14:22:43.956 INFO APP: GET /health/zwave 301 1.248 ms - 191 2024-01-17 14:23:14.106 INFO APP: GET /health/zwave 301 2.362 ms - 191 2024-01-17 14:23:44.231 INFO APP: GET /health/zwave 301 1.678 ms - 191 2024-01-17 14:24:14.343 INFO APP: GET /health/zwave 301 1.340 ms - 191 2024-01-17 14:24:44.474 INFO APP: GET /health/zwave 301 1.057 ms - 191 2024-01-17 14:25:14.601 INFO APP: GET /health/zwave 301 3.168 ms - 191 2024-01-17 14:25:44.728 INFO APP: GET /health/zwave 301 1.197 ms - 191 2024-01-17 14:26:14.854 INFO APP: GET /health/zwave 301 1.318 ms - 191 2024-01-17 14:26:44.973 INFO APP: GET /health/zwave 301 2.430 ms - 191 2024-01-17 14:26:45.877 INFO Z-WAVE: Controller status: Controller is unable to transmit 2024-01-17 14:26:49.981 INFO Z-WAVE: Controller status: Controller is Ready 2024-01-17 14:26:49.985 ERROR Z-WAVE-SERVER: Failed to send the command after 5 attempts (ZW0202) ZWaveError: Failed to send the command after 5 attempts (ZW0202) at Driver.sendMessage (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5256:23) at Driver.sendCommandInternal (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5454:28) at Driver.sendSupervisedCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5499:27) at Driver.sendCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5547:30) at Proxy.set (/opt/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:264:24) at Proxy. (/opt/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:341:17) at ZWaveNode.setValue (/opt/node_modules/zwave-js/src/lib/node/Node.ts:1150:39) at NodeMessageHandler.handle (/opt/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:19:43) at Object.node (/opt/node_modules/@zwave-js/server/dist/lib/server.js:42:92) at Client.receiveMessage (/opt/node_modules/@zwave-js/server/dist/lib/server.js:125:99) 2024-01-17 14:26:52.144 INFO Z-WAVE: Controller status: Controller is unable to transmit 2024-01-17 14:26:56.251 INFO Z-WAVE: Controller status: Controller is Ready 2024-01-17 14:26:56.254 ERROR Z-WAVE-SERVER: Failed to send the command after 5 attempts (ZW0202) ZWaveError: Failed to send the command after 5 attempts (ZW0202) at Driver.sendMessage (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5256:23) at Driver.sendCommandInternal (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5454:28) at Driver.sendSupervisedCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5499:27) at Driver.sendCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5547:30) at Proxy.set (/opt/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:264:24) at Proxy. (/opt/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:341:17) at ZWaveNode.setValue (/opt/node_modules/zwave-js/src/lib/node/Node.ts:1150:39) at NodeMessageHandler.handle (/opt/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:19:43) at Object.node (/opt/node_modules/@zwave-js/server/dist/lib/server.js:42:92) at Client.receiveMessage (/opt/node_modules/@zwave-js/server/dist/lib/server.js:125:99) 2024-01-17 14:27:15.098 INFO APP: GET /health/zwave 301 1.381 ms - 191 2024-01-17 14:27:45.211 INFO APP: GET /health/zwave 301 3.126 ms - 191 2024-01-17 14:28:15.368 INFO APP: GET /health/zwave 301 2.154 ms - 191 2024-01-17 14:28:45.489 INFO APP: GET /health/zwave 301 1.410 ms - 191 2024-01-17 14:29:15.619 INFO APP: GET /health/zwave 301 1.218 ms - 191 2024-01-17 14:29:45.748 INFO APP: GET /health/zwave 301 2.126 ms - 191 2024-01-17 14:30:10.716 INFO Z-WAVE: Controller status: Controller is unable to transmit 2024-01-17 14:30:14.824 INFO Z-WAVE: Controller status: Controller is Ready 2024-01-17 14:30:14.827 ERROR Z-WAVE-SERVER: Failed to send the command after 5 attempts (ZW0202) ZWaveError: Failed to send the command after 5 attempts (ZW0202) at Driver.sendMessage (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5256:23) at Driver.sendCommandInternal (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5454:28) at Driver.sendSupervisedCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5499:27) at Driver.sendCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5547:30) at Proxy.set (/opt/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:264:24) at Proxy. (/opt/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:341:17) at ZWaveNode.setValue (/opt/node_modules/zwave-js/src/lib/node/Node.ts:1150:39) at NodeMessageHandler.handle (/opt/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:19:43) at Object.node (/opt/node_modules/@zwave-js/server/dist/lib/server.js:42:92) at Client.receiveMessage (/opt/node_modules/@zwave-js/server/dist/lib/server.js:125:99) 2024-01-17 14:30:15.870 INFO APP: GET /health/zwave 301 1.307 ms - 191 2024-01-17 14:30:43.313 INFO Z-WAVE: Controller status: Controller is unable to transmit 2024-01-17 14:30:46.010 INFO APP: GET /health/zwave 301 1.191 ms - 191 2024-01-17 14:30:47.423 INFO Z-WAVE: Controller status: Controller is Ready 2024-01-17 14:30:47.426 ERROR Z-WAVE-SERVER: Failed to send the command after 5 attempts (ZW0202) ZWaveError: Failed to send the command after 5 attempts (ZW0202) at Driver.sendMessage (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5256:23) at Driver.sendCommandInternal (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5454:28) at Driver.sendSupervisedCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5499:27) at Driver.sendCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5547:30) at Proxy.set (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:138:24) at Proxy. (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:160:16) at ZWaveNode.setValue (/opt/node_modules/zwave-js/src/lib/node/Node.ts:1150:39) at NodeMessageHandler.handle (/opt/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:19:43) at Object.node (/opt/node_modules/@zwave-js/server/dist/lib/server.js:42:92) at Client.receiveMessage (/opt/node_modules/@zwave-js/server/dist/lib/server.js:125:99) 2024-01-17 14:30:58.934 INFO Z-WAVE: Controller status: Controller is unable to transmit 2024-01-17 14:31:03.039 INFO Z-WAVE: Controller status: Controller is Ready 2024-01-17 14:31:03.042 ERROR Z-WAVE-SERVER: Failed to send the command after 5 attempts (ZW0202) ZWaveError: Failed to send the command after 5 attempts (ZW0202) at Driver.sendMessage (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5256:23) at Driver.sendCommandInternal (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5454:28) at Driver.sendSupervisedCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5499:27) at Driver.sendCommand (/opt/node_modules/zwave-js/src/lib/driver/Driver.ts:5547:30) at Proxy.set (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:138:24) at Proxy. (/opt/node_modules/@zwave-js/cc/src/cc/BinarySwitchCC.ts:160:16) at ZWaveNode.setValue (/opt/node_modules/zwave-js/src/lib/node/Node.ts:1150:39) at NodeMessageHandler.handle (/opt/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:19:43) at Object.node (/opt/node_modules/@zwave-js/server/dist/lib/server.js:42:92) at Client.receiveMessage (/opt/node_modules/@zwave-js/server/dist/lib/server.js:125:99)

ember1205 commented 8 months ago

@ember1205 I'd love to hear your results once it's shut down. I rebooted my HA as well, not sure if that will help?

Shut down second hub, rebooted HA, problems persist.

I'll be bringing the second hub back online as it clearly made zero difference in my situation and I do still reference it for certain setup details that I want to capture / replicate.

tlm4594 commented 8 months ago

any updates? I'm constantly seeing my zwave controller getting jammed. I have 41 devices, plenty of repeaters, but having very similar issues as above... I have a zooz 700 seriers dongle, proxmox, 2024.1.5 core, 2023.12.1 Supervisor, 11.4 OS. Never had an issue with Zwave until the last couple of updates.

Polosaz commented 8 months ago

I continue to suffer, and perhaps more frequently lately, jammed receiver of short duration but high frequency. I hope we can have a solution to this problem so we can enjoy HA again as before

AlCalzone commented 8 months ago

No updates from Silabs so far, sorry. Other than "they can reproduce".

tlm4594 commented 8 months ago

Thanks for the responses, really sad this is happening. I use Z-Wave due to its supposedly stringent testing and certification process. I pay that extra price and it's been my backbone because of that (although I do use zigbee for quite a bit of devices as well, zwave has always been rock solid). I'm using zwave for my door locks and this has now become a major security issue where I've removed my door locks completely from the system due to this.

joel-bourquard commented 8 months ago

Same here, Z-Wave really deserves a timely and proper fix to this issue. I have high hopes! Thanks a lot to you @AlCalzone for following the topic with Silabs. Much appreciated 👍

jfhautenauven commented 8 months ago

I think Silicon Labs are trying to patch. Tested 7.21.0, I have the controller jammed issue.

I see on their release notes that they published a sort of downgraded version 7.19.5GA.

image

I don't see any relevant stuff in the release notes about the controller jammed though ... but i'm no expert.

https://www.silabs.com/documents/public/release-notes/SRN14910-7.19.5.0.pdf

kpine commented 8 months ago

7.19.5 is not a downgrade, it's just an update from 7.19.4. They maintain multiple versions at the same time.

https://wwwqa.silabs.com/developers/z-wave-700-series/life-cycle

AlCalzone commented 7 months ago

Not fixed in 7.21.1 either: grafik

Jgree321 commented 7 months ago

I went from never experiencing this issue to it being completely debilitating to my Zwave setup. Using zooz s2 stick on 7.18.1 firmware Only recent change I can say I made was adding an additional dimmer. Downgraded JS UI a few times with no help. Rebooted entire PC. Nothing I do seems to help. All nodes show "healthy" just can't send commands to anything essentially. Status stays jammed almost nonstop. Kinda at a loss. Thought about trying to upgrade firmware of Zooz stick.

ember1205 commented 7 months ago

I went from never experiencing this issue to it being completely debilitating to my Zwave setup. Using zooz s2 stick on 7.18.1 firmware Only recent change I can say I made was adding an additional dimmer. Downgraded JS UI a few times with no help. Rebooted entire PC. Nothing I do seems to help. All nodes show "healthy" just can't send commands to anything essentially. Status stays jammed almost nonstop. Kinda at a loss. Thought about trying to upgrade firmware of Zooz stick.

Odds are, you have at least one device in your network configured wrong and sending all kinds of meaningless information and data. The ONLY way I can get my network to be relatively stable is to shut down absolutely any and all unnecessary data being "automatically sent" from the devices to the controller.

Jgree321 commented 7 months ago

I went from never experiencing this issue to it being completely debilitating to my Zwave setup. Using zooz s2 stick on 7.18.1 firmware Only recent change I can say I made was adding an additional dimmer. Downgraded JS UI a few times with no help. Rebooted entire PC. Nothing I do seems to help. All nodes show "healthy" just can't send commands to anything essentially. Status stays jammed almost nonstop. Kinda at a loss. Thought about trying to upgrade firmware of Zooz stick.

Odds are, you have at least one device in your network configured wrong and sending all kinds of meaningless information and data. The ONLY way I can get my network to be relatively stable is to shut down absolutely any and all unnecessary data being "automatically sent" from the devices to the controller.

I'm not familiar with configuring at that level I assume it's within Zwave JS UI options within each node id?

PeteRager commented 7 months ago

First step is to identity the node. In zwavejsui click through each node and look at the RX and TX node statistics. Find the ones that are highest. Once that is figured out, then address each node one by one.

Jgree321 commented 7 months ago

First step is to identity the node. In zwavejsui click through each node and look at the RX and TX node statistics. Find the ones that are highest. Once that is figured out, then address each node one by one.

Not sure if it counts from last restart of addon or what. I do see switches such as an Inovelli Red series with 1600 TX vs other items only having a few. Where do I go to try and quiet that down?

ember1205 commented 7 months ago

First step is to identity the node. In zwavejsui click through each node and look at the RX and TX node statistics. Find the ones that are highest. Once that is figured out, then address each node one by one.

Not sure if it counts from last restart of addon or what. I do see switches such as an Inovelli Red series with 1600 TX vs other items only having a few. Where do I go to try and quiet that down?

Look at details under the "Configuration" drop-down.

PeteRager commented 7 months ago

Yes the counters reset when zwavejsui restarts.

There is a script here that can help identify the specific items.

https://community.home-assistant.io/t/determining-what-zwave-values-are-creating-the-most-traffic/528727/3

Jgree321 commented 7 months ago

Yes the counters reset when zwavejsui restarts.

There is a script here that can help identify the specific items.

https://community.home-assistant.io/t/determining-what-zwave-values-are-creating-the-most-traffic/528727/3

As an update. With nothing helping I tried safe mode and that seemed to make everything responsive once again. Now trying to isolate which addon is the issue but I'm betting on Bubble Card. I had been doing a UI overhaul recently updating dashboard with HACS addon bubble card a lot.
No idea why or how it would be crippling the Zwave network.

jgjestad commented 7 months ago

First step is to identity the node. In zwavejsui click through each node and look at the RX and TX node statistics. Find the ones that are highest. Once that is figured out, then address each node one by one.

Hello! I´m not very well fared in the details of how all of this works. I did however try to look through all my nodes as you explained.

Could you, or someone else, please try to give me some more insight into what is a lot and what is to be expected? Below are some stats I picket out from my zwavejsui now.

I have tried to go into the configuration of all my z-wave devices and increase the reporting intervals etc, to avoid too much traffic. But I wonder, might there be some that are not shown in the default configuration UI that I am missing?

Additional information:

Thanks in advance!

ember1205 commented 7 months ago

@jgjestad

Example for the Smart Switch 6 - here is one of mine where I have shut down "almost all" of the automatic reporting content and slimmed down what it DOES report:

Screen Shot 2024-02-22 at 3 33 43 PM Screen Shot 2024-02-22 at 3 33 58 PM
ember1205 commented 7 months ago

And here's my Multi Sensor 6 which I used ONLY to monitor humidity level to determine when to turn on the dehumidifier:

Screen Shot 2024-02-22 at 3 35 50 PM Screen Shot 2024-02-22 at 3 36 00 PM Screen Shot 2024-02-22 at 3 36 13 PM Screen Shot 2024-02-22 at 3 36 23 PM