Z-Wave Devices (Undefined/Undefined) after restart

[Matt-PMCT]

The problem

Either when upgrading from 0.107.6 -> 0.107.7, or when upgrading the supervisor, or just on a random restart I noticed that some of my Z-Wave device stopped working. I have almost 50 devices, and about 15 of them now showed up in the Z-Wave configuration tab as (Node:undefined undefined) instead of the normal node ids.

I tried opening/closing a door sensor and I got the following log entries in the Z-Wave log:

2020-03-31 17:08:37.245 Info, Node028, ApplicationCommandHandler - Unhandled Command Class 0x71
2020-03-31 17:08:44.511 Detail, Node028,   Received: 0x01, 0x12, 0x00, 0x04, 0x00, 0x1c, 0x0a, 0x71, 0x05, 0x00, 0x00, 0x00, 0xff, 0x06, 0x16, 0x00, 0x00, 0xbc, 0x00, 0xd8
2020-03-31 17:08:44.512 Detail,
2020-03-31 17:08:44.512 Info, Node028, ApplicationCommandHandler - Unhandled Command Class 0x71
2020-03-31 17:08:47.444 Detail, Node023,   Received: 0x01, 0x0a, 0x00, 0x04, 0x00, 0x17, 0x02, 0x98, 0x40, 0xc7, 0x00, 0xfb

I then started browsing the forums and found this thread of people having problems with Z-wave, and there was some mention of replacing the zwcfg.xml file from a backup.

I pulled my backup did a WinMerge with the existing file and I found that the existing file that particular node that I was having problems with (Node 28) had nearly all of its CommandClass tags missing in the new file. In fact, all of the nodes that I had listed as undefined undefined were missing data. Attached are my two files below I think some process is deleting data from the zwcfg.xml file in the latest versions of Home Assistant.

Environment

arch | armv7l dev | false docker | true hassio | true os_name | Linux os_version | 4.19.106-v7 python_version | 3.7.7 timezone | America/Los_Angeles version | 0.107.7 virtualenv | false

• Home Assistant Core release with the issue: V0.107.7
• Last working Home Assistant Core release (if known): V0.107.5
• Operating environment (Home Assistant/Supervised/Docker/venv): HASS
• Integration causing this issue: Zwave
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/zwave/

Problem-relevant configuration.yaml

configuration.yaml.txt

Traceback/Error logs

Additional information

HA107ZwaveXml.txt backupZwaveXml.txt

[Matt-PMCT]

I was able to restore to a backup, but the last one I had was 0.100.5. All the Z-wave device work when I use that backup. But I lost too many other things, so I have reverted back to 0.107.7 and am resigned to having to manually exclude at the unknown devices, re-add them, change all the automations that referenced the old devices and rename the several dozen associated entities...

[probot-home-assistant[bot]]

Hey there @home-assistant/z-wave, mind taking a look at this issue as its been labeled with a integration (zwave) you are listed as a codeowner for? Thanks!

[kaijk]

I too had this behavior, but in 0.106.4, this week. After a restart (that was initiated probably before zwave had finished starting from a prior restart) the zwave device states reported 'unknown' without the typical attributes. The zwcfg_* file had all of the node entries, but many of them had lost the details leaving empty or partial xml tags, or zeroed-out values. Those nodes that did have full data had their node "name" fields truncated to 16 characters.

Before restoring a backup zwcfg file, I stopped HA then renamed the zwcfg file and restarted HA for it to generate a new zwcfg file. No file was auto-created, Selecting Save Configuration created a corrupt file as noted.

I restored a backup zwcfg file and the system is working normally now.

I'm using an Aeon Gen-5 controller on a RasPI running homeassistant (nee hassio)

Here's an example node from the corrupt zwcfg file:

        <Node id="44" name="" location="" basic="0" generic="0" specific="0" type="" listening="true" frequentListening="false" beaming="false" routing="false" max_baud_rate="0" version="0" query_stage="Complete">
                <Manufacturer id="0" name="">
                        <Product type="0" id="0" name="" />
                </Manufacturer>
                <CommandClasses />
        </Node>

[Insension]

Same problem here; upon upgrading from 0.102.x to 0.107.7 all my zwave devices except the controller (Aeotec usb stick, HA running in a Freenas jail) are listed in HA as 'unknown' and the zwcfg* file contains all the nodes but these contain empty strings or zeroed out values like @kaijk mentioned above. Deleting and regenerating the zwcfg* file by restarting HA results in the same empty strings per node.

[kaijk]

@Insension try stopping HA, replace the corrupt zwcfg file with a backup, then restart, maybe more than once. Be sure that zwave has completely started before you do another restart. Just a suggestion.

[krissen]

I'm also experiencing the same issue since I upgraded to 0.107.*.

As has been mentioned earlier in the thread, it is solved by replacing the corrupt zwcfg-file with a working backup. However, it reoccurs, seemingly randomly, on HA -restarts.

On some restarts, no Zwave-devices work (all are undefined: undefined). On some restarts, some Zw-ave devices work as expected; some not (undefined: undefined). One some restarts, all Zwave-devices work. In all cases, where some or all Zwave-devices are not functional, it is solved by replacing the corrupt zwcfg -file with a working backup. But the problem keeps re-occurring.

[kpine]

Corrupted zwcfg*.xml files is a long standing issue, unlikely to be improved until HA adopts OZW 1.6. 1.6 has introduced changes to reduce the likelihood of the cache file being corrupted.

Otherwise, don't restart HA before the z-wave network completes starting and that will reduce the chance of corrupting the cache file. If the bogus node info is stored to the cache file, I don't believe OZW will try to update it, it certainly doesn't sound like it.

A general tip: if you add a new node, or remove one, always save the configuration immediately (via the control panel). This will flush the cache file to disk and in case of an HA crash or a host power cycle, the cache data will not be missing any nodes or contain removed ones.

A backup is the quickest way to fix a corrupt cache file, but it can also be manually re-created. There is no need to remove/re-add devices or hard reset usb sticks. If this happens, and you don't have a backup, you can stop HA, delete the zwcfg*.xml file, and start HA. Z-Wave will startup and re-interview the nodes, the interview process is what populates the cache file. Mains-powered (or FLIRS) will be interviewed immediately. Battery devices will not be interviewed until either a) they wake up (could be hours or days) or b) you force the device to wake up and send a NIF (node info frame) to the controller. The latter is usually achieved by following the same procedure that was used to add the device, such as pressing some button one or more times, or on some cases you can operate the device (flip switches, etc.). As a last resort, removing and adding the node accomplishes the same thing.

It's helpful to post your OZW_Log.txt file when submitting an issue. Were there errors when the network was starting? People seem to be reporting a log of CAN received...triggering resend or ERROR: Dropping command errors during startup after upgrading 0.107. If that happens, things just don't seem to work. Is there a problem in 0.107.0, or is it just coincidence from updating and restarting? This also seems to be exclusive to Docker or Hassio, curious if anyone here is not running with those. I am using HA Core with Docker, and have not had this issue.

[kaijk]

... It's helpful to post your OZW_Log.txt file when submitting an issue. Were there errors when the network was starting? People seem to be reporting a log of CAN received...triggering resend or ERROR: Dropping command errors during startup after upgrading 0.107. If that happens, things just don't seem to work. Is there a problem in 0.107.0, or is it just coincidence from updating and restarting? This also seems to be exclusive to Docker or Hassio, curious if anyone here is not running with those. I am using HA Core with Docker, and have not had this issue.

I get the CAN and Dropping Command errors on a number of devices (hassio on RasPi) on version 0.106.4. I actually got CAN errors on the controller (node-1) yesterday and zw wouldn't start at all. Gave it another try and it came up. I'm spooked and sticking to 106 for now, but I seem to recall CAN errors before 106 even.

Is there a way to rectify CAN Errors?

[kpine]

Typically both of those can be related to RF interference. The OZW knowledge base has some descriptions of the problems and possible fixes.

CAN errors: http://www.openzwave.com/knowledge-base/rfissues Dropped command errors: http://www.openzwave.com/knowledge-base/msgdropped

It could be the CAN messages are harmless in your case, I don't know how significant they are when occurring during startup. That link above seems to indicate they're not a big deal unless you've got a lot. I've just noticed they often appear in these same issues.

[benjR]

same for me zwave completely stopped working with last upgrade 0.107.*

[Matt-PMCT]

If this happens, and you don't have a backup, you can stop HA, delete the zwcfg*.xml file, and start HA. Z-Wave will startup and re-interview the nodes, the interview process is what populates the cache file. Mains-powered (or FLIRS) will be interviewed immediately. Battery devices will not be interviewed until either a) they wake up (could be hours or days) or b) you force the device to wake up and send a NIF (node info frame) to the controller.

@kpine Deleting the bad zwcfg*.cfg xml file does NOT fix this problem, the same devices remain bad when the file is regenerated, and continue to not work.

[kaijk]

If this happens, and you don't have a backup, you can stop HA, delete the zwcfg*.xml file, and start HA. Z-Wave will startup and re-interview the nodes, the interview process is what populates the cache file. Mains-powered (or FLIRS) will be interviewed immediately. Battery devices will not be interviewed until either a) they wake up (could be hours or days) or b) you force the device to wake up and send a NIF (node info frame) to the controller.

... Deleting the bad zwcfg*.cfg xml file does NOT fix this problem, the same devices remain bad when the file is regenerated, and continue to not work.

You need to restore a backup zwcfg file. You're right, the system won't create a complete one itself if a corrupt one has appeared.

[kpine]

Yes, it will create a new one. If you are having network errors though, that will prevent it from working.

[mlowijs]

Stopping hass, deleting the file and restarting it does not recreate a working zwave cfg for me. Restoring a backup makes everything work again.

[benjR]

Restored a backup too, worked for a couple of hours and stopped again

[krissen]

Is anyone else, who is suffering from the Z-wave issue, also experiencing something similar with automation (or other services)?

I've begun to see, on random restarts --- similar to the Z-wave issue --- that roughly half of my automations become unavailable. Or more specifically, in many cases (but not all) they are being duplicated. I'm seeing the original automation, say automation.myautomation, being unavailable. But a duplicate automation has (in many cases) been created, called automation.myautomation_2 which is available. However, there are also automations without duplicates, so they're simply unavailable. But also some automations which are unaffected. The issue is solved by manually removing all unavailable automations and restarting.

I'm mentioning this in case there's actually some bug elsewhere in HA core, which is causing both the Z-wave undefined: undefined error, and at the same time affecting other services, like automations.

[kaijk]

Is anyone else, who is suffering from the Z-wave issue, also experiencing something similar with automation (or other services)? ...

No, but I use yaml mode. Do you?

[krissen]

Is anyone else, who is suffering from the Z-wave issue, also experiencing something similar with automation (or other services)? ...

No, but I use yaml mode. Do you?

I use yaml-mode, yes. So when I (referring to my earlier post) restart HA, having cleaned out all the erroneously "unavailable" automations, 1) the duplicate (e.g. automation.myautomation_2) automations are gone and 2) the original ones (e.g. automation.myautomation) are available and functional...

[krissen]

I see 0.108.0b3 is out, with an update of zha dependencies.

Is anyone experiencing the troubles in this thread trying out beta versions?

[Insension]

Have not tried the beta versions yet, but see no mention of updated zwave dependencies either. ZHA stands for Zigbee Home Automation.

[krissen]

Have not tried the beta versions yet, but see no mention of updated zwave dependencies either. ZHA stands for Zigbee Home Automation.

Oh, I see. Thanks for clearing that up!

EDIT: Though there does seem to be a bump related to Zwave, too, unless I'm misunderstanding OZW as well. x-D

[benjR]

@krissen Just tried the Home Assistant 0.108.0b4 to see if anything changes but no luck, zwave not starting correctly

[geeak]

I have not been able to investigate this properly, but I noticed that my OZW log on 107.7 complains about not being able to get an exclusive lock on ttyACM0 (which is my z-wave stick). And checking the open files there are two python instances, one python and one python3 having /dev/ttyACM0 open. Starting up HA with a zwcfg restored from backup does not help.

My guess is that the two processes are clobbering each other and both end up getting partial and corrupted data, and mayhem ensues.

0.106.6 does not exhibit this issue.

Running the official docker images.

[binbin2000]

@geeak Interesting find. After upgrading I noticed that some notifications were sent twice but couldn't understand why. When looking at running processes within the docker container there are actually two python processes running as you say, which would likely explain the double notifications and the OZW problems.

I'm also running the official docker image by the way.

Edit: Just to confirm what @geeak found. After downgrading to 0.106.6 there is only one python process running in the docker container (not python3) and the duplicated notifications are gone. Removing the Z-wave integration, restoring a backup zwcfg file and adding the integration again seems to make everything work as normal again.

[binbin2000]

Could this somehow be related to PR #33168? I know little about how the dockerbuild works, but it seems to have changed files related to starting the python process within the container. But I could be wrong.

[Insension]

@binbin2000 I'm not sure if this is related to docker. I'm running core homeassistant in a freenas jail. I only have one python process (homeassistant) running.

Just to be sure I tried downgrading python-openzwave to version 0.4.18, running openzwave 1.4.3311 ; After rebooting HA the unknown devices still persisted.

Restoring a very old zwcfg*.xml file appears to bring my zwave devices back up as previous commenters have suggested. Unfortunately I have no recent backup lying around... (I know, silly me)

[kpine]

@geeak @binbin2000 That is an already known problem. https://github.com/home-assistant/core/issues/33048#issuecomment-602032390. Re-create your docker container, removing a run argument if you have one set.

@Insension HA doesn't use python-openzwave, it uses homeassistant-pyozw. Downgrading it won't do anything either because the version number is fixed in the component code. HA will just upgrade it on startup if it finds the wrong version. You'd need to modify the source code to set a different version.

[kpine]

Hmm, not sure why you can't read it. The issue is closed but definitely not locked. I just opened it in a private browser window without a problem.

Basically, if you are using Docker and have overridden CMD, then two instances of HA will start as you've indicated. The latest versions of the containers use ENTRYPOINT instead. Some container management software will do this and you might not be aware. More info here: https://github.com/home-assistant/docker/issues/107

[benjR]

https://github.com/home-assistant/core/issues/33048#issuecomment-602032390 fixed it for me. Thanks @kpine for pointing to the right direction.

[geeak]

@kpine : I can - it just turns out that I am an idiot with Chrome tabs..

It also turns out that you are 100% correct in your analysis of what's going on. I am using Synology Docker just as the other commenter, and i export/import the settings to recreate the container with a new image whenever I upgrade. It turns out that there is a"cmd" : "python -m homeassistant --config /config", in the JSON formatted settings file.

Once I changed that to "cmd" : "", it works fine. This is the solution for everyone using Synology Docker.

[daxda]

I had the usual problem and I also apparently solved it by recovering the zwcfg_ * xml file

[MarkoBolt]

@kpine : I can - it just turns out that I am an idiot with Chrome tabs..

It also turns out that you are 100% correct in your analysis of what's going on. I am using Synology Docker just as the other commenter, and i export/import the settings to recreate the container with a new image whenever I upgrade. It turns out that there is a"cmd" : "python -m homeassistant --config /config", in the JSON formatted settings file.

Once I changed that to "cmd" : "", it works fine. This is the solution for everyone using Synology Docker.

This has worked for me. :-)

[MickBru]

Hi!

I think I have the same issue, but I've no backup of the zwcfg :-( Is there any solution to rebuild this file?

Thanks!

[kaijk]

Not that I know of. But, a zwcfg backup will be in a Full Snapshot if you have one of those.

On Sat, Apr 11, 2020 at 11:09 PM MickBru notifications@github.com wrote:

Hi!

I think I have the same issue, but I've no backup of the zwcfg :-( Is there any solution to rebuild this file?

Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/33486#issuecomment-612569567, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL2THYS2CWA6E7KEJPHHIMTRMFLJFANCNFSM4LYIC27Q .

[MickBru]

Unfortunately I've setup a brand new install a few weeks ago, so I was waiting for a stable setup before doing a full snapshot or backup… I tried to delete le zwcfg file and reboot. The file is rebuild but it's not really better…

Last option seem to remove all devices from network then add them back… Hope I can find a better way…

[kaijk]

If you're starting over with zw, first take a Full Snapshot and move it off your HA machine for safekeeping...

After you remove all the zw devices you may also want to reset the controller to factory. I've yet to recover a node-id for later use following a node removal.

In Configuration>customize you can set the "Failed" property of the node device to true which will allow the zw control panel to "Remove Failed Node" although who knows what will happen or be available to do for "unknown" nodes. You'll also need to reset your devices to "factory" before re-adding them later...

Then make sure all the references to nodes and entities are gone from /config/.storage files after first looking for them in Configuration>entities & >devices and maybe removing them there. Some of the entries in .storage files have identifiers like -<string of numbers> (hyphen between nodeid and string)

Save that Snapshot in case this all goes awry. Happy Easter

On Sun, Apr 12, 2020, 1:17 PM MickBru notifications@github.com wrote:

Unfortunately I've setup a brand new install a few weeks ago, so I was waiting for a stable setup before doing a full snapshot or backup… I tried to delete le zwcfg file and reboot. The file is rebuild but it's not really better…

Last option seem to remove all devices from network then add them back… Hope I can find a better way…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/33486#issuecomment-612670260, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL2THYQMGUUUHXLQQ3CVUV3RMIOURANCNFSM4LYIC27Q .

[ljmgd8]

The problem is still there. Although it is on the HA side. When Zwave becomes unresponsive in HA, the Z-wave network still works, as zwave sensor still triggers the z-wave light. Havent try reseting everything as per kajik post. Will do that next

[kaijk]

It appears as if the zwcfg file is a necessary link for the devices to be "known". My devices and entities all looked good in the .storage registries, and the OZW log file showed everything working from device initiated events - motion devices reporting, temperature senders reporting.

But, HA front end wouldn't "see" the devices correctly without the zwcfg file. And, the zwave control panel would not create a viable zwcfg file using "save config."

So, I restored a zwcfg backup, and it all works again. Caveat, I haven't tried adding a new device yet. Hope it works when I do.

On Tue, Apr 14, 2020 at 10:47 AM ljmgd8 notifications@github.com wrote:

The problem is still there. Although it is on the HA side. When Zwave becomes unresponsive in HA, the Z-wave network still works, as zwave sensor still triggers the z-wave light. Havent try reseting everything as per kajik post. Will do that next

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/33486#issuecomment-613584768, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL2THYXDMZ7JJNJ45I3EXM3RMSOR3ANCNFSM4LYIC27Q .

[Dulcow]

I bumped from 0.108.4 to 0.108.5 and I ran into this issue. The CAN drop errors has led to some undefined devices. I must have rebooted 5 times or so and it came back to normal. Is there a way to get the Z-Wave config file creation/update fix in OZW 1.4 branch? Getting on 1.6 might not happen anytime soon.

[ljmgd8]

I removed the zw integration, reset the stick, excluded all the devices, include them back. It worked...until push for the new update available...reboot...works, then another update (this time for operating system) again not working.

[Dulcow]

Rebooted to apply OS update and it's f***** again :-( I will have to unpair and pair my smoke sensor again. This is getting very frustrating.

[Dulcow]

Restoring the OZW configuration file from a previous backup seems to have worked for me. My battery devices are the only ones not reporting yet.

I will try to follow @kpine advices when rebooting HA and adding/removing nodes.

[MickBru]

I think I’ll try to reset everything and rebuild from zero my ZW network... but I’m not very confident as I read your comments saying bug is coming again and again... Hope it will be fixed soon!

[zfubar86]

I was able to recover the listening devices by manually reconstructing the Node header and the Manufacturer section of zwcfg, however no such luck for battery operated devices.

I appreciate all the work guys but this bug is really a pita.

[Dulcow]

I was able to recover the listening devices by manually reconstructing the Node header and the Manufacturer section of zwcfg, however no such luck for battery operated devices.

I appreciate all the work guys but this bug is really a pita.

I was in the exact same situation as you are. I did stop HA while SSH'ing into the Supervisor. Then I replaced the CFG file with one that was OK from the a previous backup. Restarted HA and my battery devices were back. Have you tried this?

I think that HA must be stopped while you update the CFG file otherwise it will overwrite it when you are restarting your HA instance.

[zfubar86]

I was able to recover the listening devices by manually reconstructing the Node header and the Manufacturer section of zwcfg, however no such luck for battery operated devices. I appreciate all the work guys but this bug is really a pita.

I was in the exact same situation as you are. I did stop HA while SSH'ing into the Supervisor. Then I replaced the CFG file with one that was OK from the a previous backup. Restarted HA and my battery devices were back. Have you tried this?

I think that HA must be stopped while you update the CFG file otherwise it will overwrite it when you are restarting your HA instance.

Unfortunately I don't have a good copy of zwcfg. I have tried the same procedure I described for the listening devices but I'm not being able to refresh them to get the entities back. My next step is to physically collect them all and enable the testing mode.

[seichter]

Just to add some more noise - was just about to open a new issue. Got the same problem but when upgrading from 0.108.4 to 0.108.6 - I'm on homeassistant (former hass.io) on a Raspberry PI 3 with a Z-Wave.ME stick and about 20 Zwave devices. The Z-Wave integration yields the following:

2020-04-21 07:40:32.214 Detail, Node019,   Received: 0x01, 0x08, 0x00, 0x04, 0x00, 0x13, 0x02, 0x98, 0x40, 0x3a
2020-04-21 07:40:32.214 Info, Node019, Received SecurityCmd_NonceGet from node 19
2020-04-21 07:40:32.214 Warning, Node019, Couldn't Generate Nonce Key for Node 19
2020-04-21 07:42:00.888 Detail, Node015,   Received: 0x01, 0x0c, 0x00, 0x04, 0x00, 0x0f, 0x06, 0x31, 0x05, 0x01, 0x42, 0x07, 0x7f, 0xf1
2020-04-21 07:42:00.888 Detail,
2020-04-21 07:44:58.412 Detail, Node014,   Received: 0x01, 0x0c, 0x00, 0x04, 0x00, 0x0e, 0x06, 0x31, 0x05, 0x01, 0x42, 0x09, 0x20, 0xa1
2020-04-21 07:44:58.412 Detail,
2020-04-21 07:45:45.010 Detail, Node020,   Received: 0x01, 0x08, 0x00, 0x04, 0x00, 0x14, 0x02, 0x98, 0x40, 0x3d
2020-04-21 07:45:45.010 Info, Node020, Received SecurityCmd_NonceGet from node 20
2020-04-21 07:45:45.010 Warning, Node020, Couldn't Generate Nonce Key for Node 20
2020-04-21 07:49:31.097 Detail, Node016,   Received: 0x01, 0x0c, 0x00, 0x04, 0x00, 0x10, 0x06, 0x31, 0x05, 0x01, 0x42, 0x08, 0x91, 0x0f
2020-04-21 07:49:31.097 Detail,
2020-04-21 07:50:00.882 Detail, Node015,   Received: 0x01, 0x0c, 0x00, 0x04, 0x00, 0x0f, 0x06, 0x31, 0x05, 0x01, 0x42, 0x07, 0xab, 0x25

... which is the error for each and everyl node in my network.

[seichter]

I tried now so many variants for recovery mentioned in this thread. None of it worked for me. Any reasonable workaround how to get devices back?

[krissen]

I tried now so many variants for recovery mentioned in this thread. None of it worked for me. Any reasonable workaround how to get devices back?

Since you're saying you've tried everything in this thread, I'm assuming it includes

1. Acquiring a copy of a zwcfg...cnf file from a time when the devices were functioning.
2. Stopping HA.
3. Replacing corrupt zwcfg...cnf with the file from step (1).
4. Starting HA.

For me, the above fixes it every time. 🤷‍♂️

[zfubar86]

Unfortunately I don't have a good copy of zwcfg. I have tried the same procedure I described for the listening devices but I'm not being able to refresh them to get the entities back. My next step is to physically collect them all and enable the testing mode.

Just to update that i was also able to recover listening devices by editing the xml to add the manufacturer section and refreshing the devices while forcing them to be online.