Zooz ZST10 Z-Wave USB no longer seen by Z-Wave.js after upgrade to current HomeAssistant OS

Chef-de-IT commented 2 years ago

The problem

After upgrade to latest HomeAssistant OS, core, and Z-Wave JS, the plugin no longer sees any Z-Wave devices. In Settings > Devices > Integrations: Z-Wave JS, it says "Retrying setup: None"

All the Z-Wave devices are grayed out.

Restoring the formerly working versions of addon_core_zwave_js_0.1.70 and core_2022.8.7 did NOT fix the issue

What version of Home Assistant Core has the issue?

core-2022.9.2, Z-Wave 0.1.74

What was the last working version of Home Assistant Core?

Home Assistant OS 9.0 core_2022.9.2

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Z-Wave JS

Link to integration documentation on our website

https://www.home-assistant.io/integrations/zwave_js/

Diagnostics information

System Information

version	core-2022.9.2
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.10.5
os_name	Linux
os_version	5.15.72
arch	x86_64
timezone	America/New_York
config_dir	/config

Home Assistant Cloud

Home Assistant Supervisor

Dashboards

Recorder

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.config_entries
Source: config_entries.py:388
First occurred: 5:29:58 PM (1 occurrences)
Last logged: 5:29:58 PM
Config entry 'Z-Wave JS' for zwave_js integration not ready yet: Failed to connect: Cannot connect to host core-zwave-js:3000 ssl:default [Connect call failed ('172.30.33.0', 3000)]; Retrying in background

Additional information

No response

homeassistant commented 2 years ago

Hey there @home-assistant/z-wave, mind taking a look at this issue as it has been labeled with an integration (zwave_js) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

zwave_js documentation zwave_js source _{^{(message by IssueLinks)}}

ckirila commented 2 years ago

Same here.

MartinHjelmare commented 2 years ago

If restoring a backup doesn't help, you should try unplugging the USB device controller and rebooting the host, then plug the USB device again.

To be more clear, what did you upgrade that resulted in the problem in your experience? Was it the Z-Wave JS add-on?

lundsola commented 2 years ago

I have the same issue after upgrading to Operating system 9.2

MartinHjelmare commented 2 years ago

@lundsola so you only upgraded the Home Assistant OS to 9.2, nothing else, and then experienced the problem?

lundsola commented 2 years ago

@lundsola so you only upgraded the Home Assistant OS to 9.2, nothing else, and then experienced the problem?

Yes, I did the upgrade this morning and after that I get the following message: " Failed to connect: Cannot connect to host core-zwave-js:3000 ssl:default [Connect call failed ('172.30.33.2', 3000)] " I will try the steps you proposed above.

gbrandsma commented 2 years ago

I did experience the same issue running:

Home Assistant 2022.10.3
Home-Assistant Operating System 9.2
Z-Wave JS version: 0.1.74
On Raspberry PI4B
Zwave.Me Usb Z-Wave Controller Plus (ZME_UZB1)

This fixed it for me:

Unplug Z-wave USB dongle
Hardware reboot
Wait for Home Assistant to fully load.
Plug in Z-wave dongle
Restart Z-waveJs addon
Reload Z-wave integration.

ckirila commented 2 years ago

This fixed it for me:

Unplug Z-wave USB dongle
Hardware reboot
Wait for Home Assistant to fully load.
Plug in Z-wave dongle
Restart Z-waveJs addon
Reload Z-wave integration.

Thanks gbrandsma! This worked for me as well. That got my integration loaded and working again but my devices weren't responding until I did a network heal. But that's not a big deal, my full network is back online now with no issues.

gbrandsma commented 2 years ago

I’m still experiencing this issue. After fixing it will work for some time but fails again. Have restarted this 4 times today.

The interesting part in my Z-WaveJS log:


2022-10-12T15:45:41.808Z CNTRLR   Failed to execute controller command after 1/3 attempts. Scheduling next try i
                                  n 100 ms.
2022-10-12T15:45:41.924Z CNTRLR   The controller response indicated failure after 2/3 attempts. Scheduling next 
                                  try in 1100 ms.
2022-10-12T15:45:43.037Z CNTRLR   [Node 019] did not respond after 2/3 attempts. Scheduling next try in 500 ms.
2022-10-12T15:45:43.470Z DRIVER   Dropping message with invalid payload
2022-10-12T15:45:43.552Z CNTRLR   Failed to execute controller command after 1/3 attempts. Scheduling next try i
                                  n 100 ms.
2022-10-12T15:45:43.661Z CNTRLR   The controller response indicated failure after 2/3 attempts. Scheduling next 
                                  try in 1100 ms.
Z-Wave error ZWaveError: Failed to send the command after 3 attempts. Transmission queue full (ZW0202)
    at Driver.sendMessage (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:3990:23)
    at Driver.sendCommandInternal (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4181:28)
    at Driver.sendCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4296:15)
    at Proxy.get (/usr/src/node_modules/@zwave-js/cc/src/cc/ManufacturerSpecificCC.ts:106:24)
    at ZWaveController.getAvailableFirmwareUpdates (/usr/src/node_modules/zwave-js/src/lib/controller/Controller.ts:4523:5)
    at Function.handle (/usr/src/node_modules/@zwave-js/server/dist/lib/controller/message_handler.js:204:54)
    at Object.controller (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:38:103)
    at Client.receiveMessage (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:105:99)
    at WebSocket.<anonymous> (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:49:45)
    at WebSocket.emit (node:events:513:28) {
  code: 202,
  context: SendDataResponse {
    host: [Driver],
    type: 1,
    functionType: 19,
    payload: <Buffer 00>,
    wasSent: false
  },
  transactionSource: '    at Driver.sendMessage (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:3990:23)\n' +
    '    at Driver.sendCommandInternal (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4181:28)\n' +
    '    at Driver.sendCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4296:15)\n' +
    '    at Proxy.get (/usr/src/node_modules/@zwave-js/cc/src/cc/ManufacturerSpecificCC.ts:106:24)\n' +
    '    at ZWaveController.getAvailableFirmwareUpdates (/usr/src/node_modules/zwave-js/src/lib/controller/Controller.ts:4523:5)\n' +
    '    at Function.handle (/usr/src/node_modules/@zwave-js/server/dist/lib/controller/message_handler.js:204:54)\n' +
    '    at Object.controller (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:38:103)\n' +
    '    at Client.receiveMessage (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:105:99)\n' +
    '    at WebSocket.<anonymous> (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:49:45)\n' +
    '    at WebSocket.emit (node:events:513:28)'
}

networkerict commented 2 years ago

I also have the same issue after upgrading to Operating system 9.2 (Aeotec Z-Wave USB Stick Z-stick 7). So if there are more people experiencing the exact issue, it must be related to this upgrade.

agners commented 2 years ago

Can you try downgrading the OS to see it's indeed the OS? In the Terminal:

ha os update --version 9.0

Joeten5 commented 2 years ago

Same issue after the upgrade to OS 9.2. Using Aeotec Z-Wave USB Stick Z-stick 7. rebooted per gbrandsma’s instructions above and everything is running again. Thank you!

das-intensity commented 2 years ago

I had this issue, and the instructions that @gbrandsma noted didn't work for me. However when I went to addons -> Z-wave JS -> configuration, none of the "device" radio buttons were selected. I selected the one as per this:

Device
[ ] /dev/ttyAMA0
[ ] /dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2293321-if00
[X] /dev/serial/by-id/usb-0658_0200-if00

then clicked save. It said it needed to restart addon to apply, so I said yes, then the logs looked like success.

Best guess is that device ID changed between the upgrade, and the config had the old one.. ?

Good luck!

Full exact list of what I did:

Followed instructions as per post by @gbrandsma
Saw addon was stopped, so attempted to start, each time got error similar (but I don't think identical) to what @gbrandsma had
Disabled Z-wave JS integration
Went to addon config and selected the conbee stick (yep that was dumb selecting my zigbee stick), and starting it didn't work
Went back to addon config and selected the option as per above, upon saving it asked to restart, and I clicked yes
Looked at logs and saw everything saying success
Went back and re-enabled the Z-wave JS integration
Saw all my Z-wave devices were back

Chef-de-IT commented 2 years ago

It said it needed to restart addon to apply, so I said yes, then the logs looked like success.

das-intensity Has the fix you described above keep working for you so far? @gbrandsma updated that his fix worked for a while and then got unstuck & he had to reboot again. Thanks

das-intensity commented 2 years ago

Yep still working so far. I did have to heal the network to "recover" one of my more recently added controllers, but that might've just been because I had tried reverting home assistant and Z-wave JS to earlier versions back before I had the sensor.

To be clear, when I ran the steps I outlined above, I had already re-updated all the versions to latest.

Chef-de-IT commented 2 years ago

In my case the Z-Wave USB was still selected in the settings:

/dev/serial/by-id/usb-Silicon_Labs_CP2102N_USB_to_UART_Bridge_Controller_da1......-if00-port0

I've done the following (via UI) to get this to work:

Rolled forward all the updates (latest production core and Z-Wave JS)
Shut down the hardware
Unplugged the Z-Wave USB stick (Zooz ZST10)
Booted the hardware back up
Hot-plugged the Z-wave USB stick back in
Disabled Z-Wave JS plugin
In Z-Wave JS plugin settings, saw that the apparently correct Z-Wave USB stick (/dev/serial/by-id/usb-Silicon_Labs.....) is still selected by ID.
Picked another random /dev/ option, saved, then picked this same selected Z-Wave USB stick (/dev/serial/by-id/usb-Silicon_Labs.....) and clicked Save again.
Re-enabled Z-Wave JS plugin, started it, specified "start on system startup", enabled Watchdog, Saved.
The thing "magically" started working again.

Needless to say, would like to see an authoritative root-cause insight / solution we can trust with confidence - as this was just poking around semi-blind.

Chef-de-IT commented 2 years ago

Update: some sections of UI (incl dashboards, config screens) hang / respond slowly. Not sure if this is due to watchdog or anything else. Will roll back the entire thing.

MartinHjelmare commented 2 years ago

I'd like all participants in this issue so far to check their controller path in the Z-Wave JS add-on configuration and post what the path to the device is in a comment. See example image below.

addon_controller_path

gbrandsma commented 2 years ago

gbrandsma commented 2 years ago

Currently my setup is running for a day without issues. In the logs I posted earlier Z-waveJS warned for "Transmission queue full". I temporarily removed a node that was sending a lot of traffic.

Do other people see this in their log? ("Transmission queue full ")

Chef-de-IT commented 2 years ago

agners commented 2 days ago

Can you try downgrading the OS to see it's indeed the OS? In the Terminal: ha os update --version 9.0

Just did it. Confirmed the problem has gone away once back on 9.0 (even with z-wave js, core up to date per prod channel)

Chef-de-IT commented 2 years ago

I'd like all participants in this issue so far to check their controller path in the Z-Wave JS add-on configuration and post what the path to the device is in a comment. See example image below.

Martin: /dev/serial/by-id/usb-Silicon_Labs_CP2102N_USB_to_UART_Bridge_Controller_da13d......68c-if00-port0 (the dots ..... above just replace a long alphanumeric string - not sure if security related)

brkane commented 2 years ago

I hit this today updating to OS 9.2, Following these steps fixed it https://github.com/home-assistant/core/issues/80146#issuecomment-1278181914

Before taking those steps my log error was slightly different

Logger: homeassistant.components.zwave_js
Source: components/zwave_js/addon.py:384
Integration: Z-Wave ([documentation](https://www.home-assistant.io/integrations/zwave_js), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+zwave_js%22))
First occurred: 7:54:20 PM (3 occurrences)
Last logged: 7:54:43 PM
Failed to set the Z-Wave JS add-on options: not a valid value for dictionary value @ data['options']. Got {'device': '/dev/serial/by-id/usb-Silicon_Labs_Zooz_ZST10_700_Z-Wave_Stick_HEXSTRING2-if00-port0', 's0_legacy_key': 'KEY', 's2_access_control_key': 'KEY', 's2_authenticated_key': 'KEY', 's2_unauthenticated_key': 'KEY'}

My device is at /dev/serial/by-id/usb-Silicon_Labs_Zooz_ZST10_700_Z-Wave_Stick_HEXSTRING-if00-port0

RayatHome commented 2 years ago

I'm having similar issues with Z-Wave JS and the Integration stopping.

Home Assistant 2022.10.4 Supervisor 2022.10.0 Operating System 9.2 Frontend 20221010.0 - latest

I'm using a Aeotec Z-Stick Gen5 This had been working for many months prior to upgrade

Options

Device

/dev/ttyAMA0

/dev/serial/by-id/usb-0658_0200-if00

phendrank commented 2 years ago

just reinstall again

Chef-de-IT commented 2 years ago

just reinstall again

That won't work for multiple, sizeable installations.

Downgrading Operating System to 9.0 works.

MartinHjelmare commented 2 years ago

@RayatHome which device was selected in the Z-Wave JS add-on configuration? /dev/ttyAMA0 or /dev/serial/by-id/usb-0658_0200-if00?

RayatHome commented 2 years ago

@RayatHome which device was selected in the Z-Wave JS add-on configuration? /dev/ttyAMA0 or /dev/serial/by-id/usb-0658_0200-if00?

@MartinHjelmare it was /dev/serial/by-id/usb-0658_0200-if00

Chef-de-IT commented 2 years ago

@MartinHjelmare I'd like to clarify that the OS 9.2 Z-Wave issue wasn't limited to USB stick being de-selected - even when as the result of the steps in https://github.com/home-assistant/core/issues/80146#issuecomment-1278181914 the correct option was picked, the entire HA UI would sometimes freeze or have hiccups, and scenes or automations that involve 10 or more Z-Wave devices (some in dimmer groups) would just hang. Mind you, this is on an i7 Dell OptiPlex with 32GB RAM running nothing but HA off latest Samsung SSD.

Rolling back to OS 9.0 fixed the hanging issue instantly.

janneman001 commented 2 years ago

I'm having similar issues with Z-Wave JS and the Integration becoming very unstable.

Home Assistant 2022.10.4 Supervisor 2022.10.0 Operating System 9.2 Frontend 20221010.0 - latest Zwave-JS addon 0.1.74 (/dev/serial/by-id/usb-0658_0200-if00) I'm using a Z-Wave.Me UZB1 stick

This had been relatively stable for months prior to upgrade. I have 23 zwave devices. It started with non-zwaveplus devices not coming on-line after the restart. Eventually some came back. I have a cripple zwave setup now. This was not the case before.

Downgrade the OS to 9.0 did help a little. It does not look like the complet solution for this.

janneman001 commented 2 years ago

Went back to pre-core update. I am now back on OS 9.0 and core 9.7. For now the system works as before and looks like either core or os upgrade did something. But as @gbrandsma experienced i wait a little with conclusion.

janneman001 commented 2 years ago

The situation is still stable. I experienced no more problems after reverting back to OS 9.0 as well as core 9.7 6 days ago. Also the versions of zwave-js, the add-on and the integration are the same as when i was on OS 9.2/ core 10.5. To be specific: Zwave-js add-on: 0.1.74, Integration: Driverversion: 10.3.0, Serverversion: 1.24.0 Some incompatibility must have come-up in either OS or core or a combination of both. From the incompatibility it looks like non-zwaveplus devices are hit the most. I hope it helps.

TheRealWaldo commented 2 years ago

Same issues as others have reported here.

I have a Zooz S2 on an i7 NUC with 32GB of RAM that was performing flawlessly for quite some time.

Now all sorts of things like it randomly starting inclusion without any request to do so, random unrelated entities appearing on devices, extremely slow and sometimes no responses from devices.

The same errors others have witnessed appear in my logs. Sometimes unplugging and plugging the adapter back in helps for a short period of time.

I have hundreds of devices with many customizations. Last time I rebuilt the network it took over a week to get everything excluded and re-added.

I may have to consider rolling back as well.

dvemail commented 2 years ago

just adding $0.02 here. Same problem for me, rolling back to 9.0 is the only thing that worked to fix it.

agners commented 2 years ago

@dvemail what is the version of the other components you are using (Core Z-Wave JS add-on)?

Did you try HAOS 9.3?

When the problem appears, can you check (and post) the host logs? I wonder if USB errors appear.

dvemail commented 2 years ago

zwave driver 10.3.0, server 1.24.0 Home Assistant 2022.10.5 Supervisor 2022.10.0 Operating System 9.3 Frontend 20221010.0 - latest

ZST10-700 by Silicon Labs Firmware: 7.17.2

"add-on is stopped' I never stopped it, and auto-start is on in status - "retrying setup: none"

2022-10-30T19:38:46.560Z DRIVER Failed to initialize the driver: ZWaveError: Timeout while waiting for an ACK from the controller (ZW0200) at Driver.sendMessage (/usr/src/node_modules/zwave-js/src/lib/driver/Drive r.ts:3990:23) at ZWaveController.interview (/usr/src/node_modules/zwave-js/src/lib/contr oller/Controller.ts:874:23) at Driver.initializeControllerAndNodes (/usr/src/node_modules/zwave-js/src /lib/driver/Driver.ts:1270:4) at Immediate. (/usr/src/node_modules/zwave-js/src/lib/driver/Dr iver.ts:1015:5) Error in driver ZWaveError: Failed to initialize the driver: ZWaveError: Timeout while waiting for an ACK from the controller (ZW0200) at Driver.sendMessage (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:3990:23) at ZWaveController.interview (/usr/src/node_modules/zwave-js/src/lib/controller/Controller.ts:874:23) at Driver.initializeControllerAndNodes (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:1270:4) at Immediate. (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:1015:5) (ZW0100) at Immediate. (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:1032:6) at processTicksAndRejections (node:internal/process/task_queues:96:5) { code: 100, context: undefined, transactionSource: undefined

On Mon, Oct 31, 2022 at 3:46 AM Stefan Agner @.***> wrote:

@dvemail https://github.com/dvemail what is the version of the other components you are using (Core Z-Wave JS add-on)?

Did you try HAOS 9.3?

When the problem appears, can you check (and post) the host logs? I wonder if USB errors appear.

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/80146#issuecomment-1296908766, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGBODBVSZFR6PHZ75752SVTWF6PQJANCNFSM6AAAAAARCYML4M . You are receiving this because you were mentioned.Message ID: @.***>

janneman001 commented 2 years ago

Gave it another try tuesday. Went from OS9.0 -> OS9.3 and core to 10.5 I only received errors at startup. I reconnected the usb-stick a few minutes after start-up. After that zwave-js started normally.

Today still without problems. To me this looks like some inrush problem with zwave.
Maybe the number of devices, integrations or addons of the install has something to do with this zwave strangeness.

Chef-de-IT commented 2 years ago

Gave it another try tuesday. Went from OS9.0 -> OS9.3 and core to 10.5 I only received errors at startup. I reconnected the usb-stick a few minutes after start-up. After that zwave-js started normally.

Today still without problems. To me this looks like some inrush problem with zwave. Maybe the number of devices, integrations or addons of the install has something to do with this zwave strangeness.

If you power-cycle the entire device (i.e. without unplugging, waiting, and plugging things back in), does it work or you have to unplug, wait, and plug back in every time the host hardware is rebooted?

janneman001 commented 2 years ago

Gave it another try tuesday. Went from OS9.0 -> OS9.3 and core to 10.5 I only received errors at startup. I reconnected the usb-stick a few minutes after start-up. After that zwave-js started normally. Today still without problems. To me this looks like some inrush problem with zwave. Maybe the number of devices, integrations or addons of the install has something to do with this zwave strangeness.

If you power-cycle the entire device (i.e. without unplugging, waiting, and plugging things back in), does it work or you have to unplug, wait, and plug back in every time the host hardware is rebooted?

I used hassio.host_shutdown, completely powered off and on again. Everything came on-line and worked. No errors in the log. NB i have a different z-wave stick than the one you have.

ipxw commented 1 year ago

I had an issue with a specific device, Zen17 relay, after updating to OS 9.2 and Core 12.6. I downgraded Core back to 12.1 via backup and then it immediately worked. So something between 12.1 and 12.6 broke something and I don't want to upgrade again to troubleshoot.

I tried healing, excluding, factory reset, every single thing, and the device kept being marked as dead after a few min. The moment I downgraded it was stable again, nothing else changed.

So specifically, 12.1 -> 12.6: device broke 12.6 -> restore 12.1 backup: device works

theoo commented 1 year ago

I had a similar problem after updating to 2023.3.1. My setup uses two dongles, one for Z-Wave and another for Zigbee, communicating with a serial port like /dev/ACM0 and /dev/ACM1 respectivelly. After a reboot serial ports mixed up and services were communicating with the wrong devices...

Restarting HA and pluging dongles in the right order fixes the issue... until the next unplaned restart.

HA doesn't seem to use udev which means to keep serial port names persistant after reboot, one needs an unique device path like /dev/serial/by-id/usb-1324-5678-WHATEVER-THE-DRIVER-SAYS

For Z-wave instead of:

I use:

And ZHA already uses a persistant path (found in core.config_entries). Updating my naive setup seems to fix the issue with both services.

issue-triage-workflows[bot] commented 1 year ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

Chef-de-IT commented 1 year ago

I had a similar problem after updating to 2023.3.1. [...] HA doesn't seem to use udev which means to keep serial port names persistant after reboot, one needs an unique device path like /dev/serial/by-id/usb-1324-5678-WHATEVER-THE-DRIVER-SAYS

For Z-wave instead of:

I use:

And ZHA already uses a persistant path (found in core.config_entries). Updating my naive setup seems to fix the issue with both services.

In my case, the USB dongle path picked by Z-Wave was the specific one. The issue doesn't appear to be happening anymore, however, on the current (as of this writing) HAOS, Core, Z-Wave

home-assistant / core