search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
72.18k stars 30.21k forks source link

New update of zwave-js and zwavejs2mqtt broke the motion detection of Fibaro FGMS001 #50598

Closed f45tb00t closed 3 years ago

f45tb00t commented 3 years ago

The problem

The motion and tampering sensors of my Fibaro FGMS001 are not updated anymore. Therefore my automations do not work. I haven't changed anything with the sensors nor the automations. The status is stuck of the sensors, either way off or on all the time.

What is version of Home Assistant Core has the issue?

core-2021.5.3

What was the last working version of Home Assistant Core?

core-2021.4.6

What type of installation are you running?

Home Assistant Core

Integration causing the issue

zwave-js, zwavejs2mqtt

Link to integration documentation on our website

https://www.home-assistant.io/integrations/zwave_js/

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Nothing special.

2021-05-14 03:45:54 WARNING (Recorder) [homeassistant.components.recorder.migration] Database is about to upgrade. Schema version: 14
2021-05-14 03:45:58 ERROR (MainThread) [homeassistant.components.zwave_js] Failed to connect: Server disconnected
2021-05-14 03:45:58 WARNING (MainThread) [homeassistant.config_entries] Config entry 'Z-Wave JS' for zwave_js integration not ready yet: Server disconnected; Retrying in background
2021-05-14 04:05:59 WARNING (Thread-5) [homeassistant.components.mqtt] Disconnected from MQTT server 127.0.0.1:1883 (1)

But it is reachable. 

sudo netstat -tulpena | grep :3000
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      0          22031      1201/docker-proxy   
tcp        0      0 172.18.0.1:34694        172.18.0.2:3000         VERBUNDEN   0          23325      1209/docker-proxy   
tcp6       0      0 :::3000                 :::*                    LISTEN      0          21064      1209/docker-proxy   
tcp6       0      0 ::1:56054               ::1:3000                VERBUNDEN   998        23888      1857/python3.8      
tcp6       0      0 ::1:3000                ::1:56054               VERBUNDEN   0          25716      1209/docker-proxy

Additional information

I already tried to downgrade. But unfortunately without real luck. Went back to v4.0.0 Also I've tried to remove all devices and readd them again. Same with the Stick ZW090, I did a complete factory reset and configured everything from scratch again. Only thing I've changed so far is that I added some new gosund sp111 power plugs. But this is 2.4 GHz and should not interfere at all with zwave on 868.4MHz. I've tried it as well with "Re-interview Node" and "Heal Network/Heal Node" Furthermore when remove and readd the device in my living room they work for a short period of time, but when placed back to their usual place 1 sensor works, the others don't.

Any help is much appreciated. Thanks you in advance.

cheers

fb

probot-home-assistant[bot] commented 3 years ago

Hey there @home-assistant/z-wave, mind taking a look at this issue as its been labeled with an integration (zwave_js) you are listed as a codeowner for? Thanks! (message by CodeOwnersMention)

MartinHjelmare commented 3 years ago

Please create a dump of your Z-Wave network and upload the dump as a text file here.

The dump tool is available in the configuration panel of the Z-Wave JS integration, reached from the integrations page in the Home Assistant GUI.

The network dump will help us troubleshoot your device.

Thanks!

f45tb00t commented 3 years ago

Hi @MartinHjelmare,

can you tell me what is included in the dump? Like the network key?

kr fastboot

MartinHjelmare commented 3 years ago

The network key is not included. The dump contains version and state information of the network, eg driver version, server version, driver options, controller and node state. It does include the home id which is unique to your serial device. We're mostly interested in the node state, so you can cut that part, for the node in question out, and paste into a new file if you prefer.

You can review all the information before you post here.

f45tb00t commented 3 years ago

Hi @MartinHjelmare,

find the dump attached.

dump210514.txt

cheers

fb

Edit: Two of the three sensors worked in the morning. Actually Im checking the logs about their status for the given sensors in the logs. Because yesterday after the update, I just realized that the motion detection never went back to "off". Same when I moved the device and the tampering sensor went "on". For all devices it should go back to "off" after ~30 seconds usually.

MartinHjelmare commented 3 years ago

Do you see duplicates of the motion and tamper sensors but with a different entity_id in the list of entities in the GUI? Please also check that the entity_id of all of these sensors are what you expect and have set in your automations.

I'm trying to rule out or confirm that a migration flow has changed the ids of these entities in one way or another.

MartinHjelmare commented 3 years ago

Also, do you have other motion sensors that are still working?

f45tb00t commented 3 years ago

Hi,

yes I saw duplicates. Until I reinstalled everything from scratch. But GUI we talk about zwavejs2mqtt, right? Coze in HA I did not see that at all. In zwavejs2mqtt I saw for example a sensor twice with the same ID.

In HA the entities have all different names. I changed this some day to have no confusion when using the sensors. All of the sensors have the room included in their IDs, just to have it unique. e.g: binary_sensor.fibaro_sensor_kueche_home_security_motion_detection

Yes I have other sensors, but not z-wave. Others are zigbee and connected to a sonoff bridge.

Note: I've used "Configuration - Device". Selected one of the sensors and checked all entities. The motion entity was stuck at "on". I went to that device and shaked it, usually the temper sensor should go on, but it stuck on "off" Then I used the dev tools and checked for the entity itself. Also there it was just "on". And it did not change it state back to "off".

Edit: Confirmed. Other zigbee motion sensors work.

MartinHjelmare commented 3 years ago

I'm only talking about the Z-Wave JS integration GUI, not zwavejs2mqtt. The latter is not relevant for the integration.

I meant to ask if you have other motion sensors in Z-Wave JS that are working.

Sorry for the confusion.

This sounds like a problem with the device in question. I think we need driver debug logs to confirm that updates are sent or not, when the device is triggered by motion. You can see and copy driver debug logs from the Z-Wave JS integration configuration panel in the GUI.

f45tb00t commented 3 years ago

Again the question. No sensitive data included in this?

MartinHjelmare commented 3 years ago

There shouldn't be sensitive data, but zwave-js is a different project so please review the log before posting here.

f45tb00t commented 3 years ago

Sorry, I am lost. Where to get the driver information? HA - Integrations - Config and then?

MartinHjelmare commented 3 years ago

It should look like this:

zwave_js_driver_logs

f45tb00t commented 3 years ago 
Abonniert Z-Wave JS-Protokollnachrichten ...
2021-05-14T13:03:37.953Z SERIAL « 0x0108000400060298402f                                              (10 bytes)
2021-05-14T13:03:37.957Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:03:37.958Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCNonceGet]
2021-05-14T13:03:37.965Z SERIAL » 0x01110013060a988093af7a7872b2a3e905edb5                            (19 bytes)
2021-05-14T13:03:37.965Z DRIVER » [Node 006] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      237
                                  └─[SecurityCCNonceReport]
                                      nonce: 0x93af7a7872b2a3e9
2021-05-14T13:03:37.968Z SERIAL « [ACK]                                                                   (0x06)
2021-05-14T13:03:37.974Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-14T13:03:37.975Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:03:37.975Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-14T13:03:37.990Z SERIAL « 0x01070013ed00000204                                                 (9 bytes)
2021-05-14T13:03:37.991Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:03:37.992Z DRIVER « [REQ] [SendData]
                                    callback id:     237
                                    transmit status: OK
2021-05-14T13:03:38.011Z SERIAL « 0x0124000400061e98818be9a02893a5bfcce50370d644160b81bb32dc934e6fd6a (38 bytes)
                                  1dac8bdbd6b
2021-05-14T13:03:38.012Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:03:38.014Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCCommandEncapsulation]
                                    │ sequenced: false
                                    └─[NotificationCCReport]
                                        notification type:   7
                                        notification status: 255
                                        notification event:  0x00
                                        event parameters:    
2021-05-14T13:03:38.017Z CNTRLR   [Node 006] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 8 => 0
2021-05-14T13:04:50.926Z SERIAL « 0x0108000400050298402c                                              (10 bytes)
2021-05-14T13:04:50.928Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:04:50.929Z DRIVER « [Node 005] [REQ] [ApplicationCommand]
                                  └─[SecurityCCNonceGet]
2021-05-14T13:04:50.940Z SERIAL » 0x01110013050a9880c96c914dbaa5fd7e05eee4                            (19 bytes)
2021-05-14T13:04:50.940Z DRIVER » [Node 005] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      238
                                  └─[SecurityCCNonceReport]
                                      nonce: 0xc96c914dbaa5fd7e
2021-05-14T13:04:50.943Z SERIAL « [ACK]                                                                   (0x06)
2021-05-14T13:04:50.949Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-14T13:04:50.949Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:04:50.950Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-14T13:04:50.965Z SERIAL « 0x01070013ee00000207                                                 (9 bytes)
2021-05-14T13:04:50.966Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:04:50.967Z DRIVER « [REQ] [SendData]
                                    callback id:     238
                                    transmit status: OK
2021-05-14T13:04:50.985Z SERIAL « 0x0123000400051d988193a593ebe4e1759fac04d67dd6a2841f68b5c9344bfed30 (37 bytes)
                                  741bee3c9
2021-05-14T13:04:50.986Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:04:50.988Z DRIVER « [Node 005] [REQ] [ApplicationCommand]
                                  └─[SecurityCCCommandEncapsulation]
                                    │ sequenced: false
                                    └─[NotificationCCReport]
                                        notification type:   Home Security
                                        notification status: 255
                                        notification state:  Motion detection
2021-05-14T13:04:50.990Z CNTRLR   [Node 005] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 0 => 8
2021-05-14T13:04:54.808Z SERIAL « 0x0108000400060298402f                                              (10 bytes)
2021-05-14T13:04:54.810Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:04:54.811Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCNonceGet]
2021-05-14T13:04:54.819Z SERIAL » 0x01110013060a9880578e98d78db5f51d05ef45                            (19 bytes)
2021-05-14T13:04:54.820Z DRIVER » [Node 006] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      239
                                  └─[SecurityCCNonceReport]
                                      nonce: 0x578e98d78db5f51d
2021-05-14T13:04:54.822Z SERIAL « [ACK]                                                                   (0x06)
2021-05-14T13:04:54.828Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-14T13:04:54.828Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:04:54.829Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-14T13:04:54.845Z SERIAL « 0x01070013ef00000206                                                 (9 bytes)
2021-05-14T13:04:54.846Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:04:54.846Z DRIVER « [REQ] [SendData]
                                    callback id:     239
                                    transmit status: OK
2021-05-14T13:04:54.865Z SERIAL « 0x0123000400061d9881ae7cb4a846caa920680dfc497d2d1421ad345753533687c (37 bytes)
                                  7e825b362
2021-05-14T13:04:54.868Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:04:54.869Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCCommandEncapsulation]
                                    │ sequenced: false
                                    └─[NotificationCCReport]
                                        notification type:   Home Security
                                        notification status: 255
                                        notification state:  Motion detection
2021-05-14T13:04:54.871Z CNTRLR   [Node 006] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 0 => 8
2021-05-14T13:05:21.786Z SERIAL « 0x0108000400050298402c                                              (10 bytes)
2021-05-14T13:05:21.787Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:05:21.790Z DRIVER « [Node 005] [REQ] [ApplicationCommand]
                                  └─[SecurityCCNonceGet]
2021-05-14T13:05:21.797Z SERIAL » 0x01110013050a9880818b7d026a395f0105f067                            (19 bytes)
2021-05-14T13:05:21.798Z DRIVER » [Node 005] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      240
                                  └─[SecurityCCNonceReport]
                                      nonce: 0x818b7d026a395f01
2021-05-14T13:05:21.800Z SERIAL « [ACK]                                                                   (0x06)
2021-05-14T13:05:21.806Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-14T13:05:21.806Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:05:21.807Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-14T13:05:21.823Z SERIAL « 0x01070013f000000219                                                 (9 bytes)
2021-05-14T13:05:21.823Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:05:21.824Z DRIVER « [REQ] [SendData]
                                    callback id:     240
                                    transmit status: OK
2021-05-14T13:05:21.843Z SERIAL « 0x0124000400051e988182a850eb2fb7c229c1b2d3e6da4ff5b84e5dbb81bf754cc (38 bytes)
                                  a459997b639
2021-05-14T13:05:21.844Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:05:21.846Z DRIVER « [Node 005] [REQ] [ApplicationCommand]
                                  └─[SecurityCCCommandEncapsulation]
                                    │ sequenced: false
                                    └─[NotificationCCReport]
                                        notification type:   7
                                        notification status: 255
                                        notification event:  0x00
                                        event parameters:    
2021-05-14T13:05:21.848Z CNTRLR   [Node 005] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 8 => 0
2021-05-14T13:05:25.989Z SERIAL « 0x0108000400060298402f                                              (10 bytes)
2021-05-14T13:05:25.990Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:05:25.992Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCNonceGet]
2021-05-14T13:05:25.998Z SERIAL » 0x01110013060a9880c1f9748a2d9fdde905f15d                            (19 bytes)
2021-05-14T13:05:25.999Z DRIVER » [Node 006] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      241
                                  └─[SecurityCCNonceReport]
                                      nonce: 0xc1f9748a2d9fdde9
2021-05-14T13:05:26.002Z SERIAL « [ACK]                                                                   (0x06)
2021-05-14T13:05:26.008Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-14T13:05:26.008Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:05:26.009Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-14T13:05:26.024Z SERIAL « 0x01070013f100000218                                                 (9 bytes)
2021-05-14T13:05:26.025Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:05:26.026Z DRIVER « [REQ] [SendData]
                                    callback id:     241
                                    transmit status: OK
2021-05-14T13:05:26.044Z SERIAL « 0x0124000400061e9881eb4d042d597d4fe8c99712ef01c8c75c5320eac19e29477 (38 bytes)
                                  dfa5f507073
2021-05-14T13:05:26.050Z SERIAL » [ACK]                                                                   (0x06)
2021-05-14T13:05:26.051Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCCommandEncapsulation]
                                    │ sequenced: false
                                    └─[NotificationCCReport]
                                        notification type:   7
                                        notification status: 255
                                        notification event:  0x00
                                        event parameters:    
2021-05-14T13:05:26.053Z CNTRLR   [Node 006] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 8 => 0
f45tb00t commented 3 years ago

Now it seems, it got the motion again for two for the sensors. For the third one in the kitchen I did not receive anything.

Bedroom - OK (only one which continued working all the time, beside when I configured everything from scratch this one was also not working) Hallway - OK (by now, yesterday evening and this night not) Kitchen - NOK (it's the nearest sensor to the usb stick)

MartinHjelmare commented 3 years ago

When you write NOK, does that mean that it does not generate a log entry when you trigger the sensor by walking in front of it eg?

If that's the case the problem lies with the device or communication between device and serial controller and probably is not related to the integration and may also not be related to the driver software (Z-Wave JS).

f45tb00t commented 3 years ago

Exactly, no log entries. But in fact all three worked like a charm for month now. After the upgrade yesterday mid day it stopped working for both of them. Surely this could happen to one device, but the chance that two devices fail at the same time is very rare. Especially what very weird was, that for example the temper sensor(seismic detector) worked somehow but the motion detection did not as it just stucked. And then the temper sensor also stucked. All of the sensors, nor the configuration were touched for weeks now. Since I've just migrated the setup to a new device. This time with zwave-js instead of ozw.

By the way: Thanks a lot for the outstanding work. Even my doorlock is now working just perfect. With ozw it did not. By the way_2: I tried this night to include all devices again when restarting from scratch. I realized that the secure integration did not work. At least in Z-Wave JS it did not. Because in zwavejs2mqtt the sensors were shown as "secure: no" So I just removed them and added the devices again via zwavejs2mqtt this time. Then it was shown "secure: yes" But yes, that would be a different ticket i guess :)

MartinHjelmare commented 3 years ago

It could still be some issue with the driver (Z-Wave JS), I don't know that well enough to rule out, but we have ruled out the integration at this point. The integration only consumes messages from the driver. So without messages, the integration won't report anything.

@AlCalzone can we rule out the driver too if there are no debug log messages when the sensor is triggered?

blhoward2 commented 3 years ago

There shouldn't be sensitive data, but zwave-js is a different project so please review the log before posting here.

We don’t include anything sensitive in the log. The home id is not meant to be sensitive because you’d need to know your address and if we knew that it’s sent in the clear over the air with every message (as part of the zwave protocol, not by us) so it would take about 10 seconds to ascertain.

AlCalzone commented 3 years ago

@AlCalzone can we rule out the driver too if there are no debug log messages when the sensor is triggered?

If there is no log at all, not even a SERIAL « when the sensor is triggered, the message does not reach the controller.

deku-m commented 3 years ago

I am facing the same problem with Neo Coolcam Motion sensor registered as Shenzhen Neo Electronics Co., Ltd. | PIR Motion Sensor with Temperature Sensor | PD03Z. After update it won't work anymore in the beginnen they did after a while they stop, looks like staying in sleep mode (in controller) they stay off for communicating with lights. But motion looks triggered.

Should i report a new issue?

MartinHjelmare commented 3 years ago

Please open a new issue and post network dump and driver debug log when triggering the motion sensor.

f45tb00t commented 3 years ago

Hello @MartinHjelmare @AlCalzone,

the sensor in the kitchen came back to life. I tried it again with heal network. Next morning 4 devices were healed.

Unfortunately yesterday evening the status was again stuck of the one in the kitchen. Now in the morning the status went back from detected to clear... I made a dump of it this time. Also will try to make a dump later and see if the status changes back to no motion and give you more logs. By the way. The time is wrong (-2h) of my real CEST time here. But the system time of the machine where HA is running is fine. Mo 17. Mai 08:13:17 CEST 2021

Abonniert Z-Wave JS-Protokollnachrichten ...
2021-05-17T06:02:19.099Z SERIAL « 0x0108000400060298402f                                              (10 bytes)
2021-05-17T06:02:19.103Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T06:02:19.104Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCNonceGet]
2021-05-17T06:02:19.114Z SERIAL » 0x01110013060a9880c6c87c664956bec3054bd1                            (19 bytes)
2021-05-17T06:02:19.115Z DRIVER » [Node 006] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      75
                                  └─[SecurityCCNonceReport]
                                      nonce: 0xc6c87c664956bec3
2021-05-17T06:02:19.118Z SERIAL « [ACK]                                                                   (0x06)
2021-05-17T06:02:19.123Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-17T06:02:19.124Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T06:02:19.125Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-17T06:02:19.140Z SERIAL « 0x010700134b000002a2                                                 (9 bytes)
2021-05-17T06:02:19.141Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T06:02:19.141Z DRIVER « [REQ] [SendData]
                                    callback id:     75
                                    transmit status: OK
2021-05-17T06:02:19.302Z SERIAL « 0x0123000400061d9881672f5c2355f3eff50f3eb7184f58204cc2ddc6e84455056 (37 bytes)
                                  9e4a6fa40
2021-05-17T06:02:19.307Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T06:02:19.310Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCCommandEncapsulation]
                                    │ sequenced: false
                                    └─[NotificationCCReport]
                                        notification type:   Home Security
                                        notification status: 255
                                        notification state:  Motion detection
2021-05-17T06:02:19.312Z CNTRLR   [Node 006] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 0 => 8
2021-05-17T06:02:49.767Z SERIAL « 0x0108000400060298402f                                              (10 bytes)
2021-05-17T06:02:49.768Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T06:02:49.770Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCNonceGet]
2021-05-17T06:02:49.780Z SERIAL » 0x01110013060a9880b1d254de2e9dc8c1054cf3                            (19 bytes)
2021-05-17T06:02:49.781Z DRIVER » [Node 006] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      76
                                  └─[SecurityCCNonceReport]
                                      nonce: 0xb1d254de2e9dc8c1
2021-05-17T06:02:49.783Z SERIAL « [ACK]                                                                   (0x06)
2021-05-17T06:02:49.789Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-17T06:02:49.790Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T06:02:49.791Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-17T06:02:49.806Z SERIAL « 0x010700134c000002a5                                                 (9 bytes)
2021-05-17T06:02:49.806Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T06:02:49.807Z DRIVER « [REQ] [SendData]
                                    callback id:     76
                                    transmit status: OK
2021-05-17T06:02:49.925Z SERIAL « 0x0124000400061e98811940085bcf5321430ecabe34141bb8db93da7ab134cb289 (38 bytes)
                                  540024b8f4e
2021-05-17T06:02:49.926Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T06:02:49.927Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCCommandEncapsulation]
                                    │ sequenced: false
                                    └─[NotificationCCReport]
                                        notification type:   7
                                        notification status: 255
                                        notification event:  0x00
                                        event parameters:    
2021-05-17T06:02:49.931Z CNTRLR   [Node 006] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 8 => 0
f45tb00t commented 3 years ago

Hi,

it happened again. Just had an eye on it. The motion status is stuck again. entity: binary_sensor.fibaro_sensor_kueche_home_security_motion_detection = ON (and does not go back to OFF) I will make now a debug log and do some movements and add the output of the logfile here as well.

Abonniert Z-Wave JS-Protokollnachrichten ...
2021-05-17T08:12:29.590Z SERIAL « 0x0108000400060298402f                                              (10 bytes)
2021-05-17T08:12:29.592Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:12:29.594Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCNonceGet]
2021-05-17T08:12:29.605Z SERIAL » 0x01110013060a98808f397cb0a921f9320575a0                            (19 bytes)
2021-05-17T08:12:29.606Z DRIVER » [Node 006] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      117
                                  └─[SecurityCCNonceReport]
                                      nonce: 0x8f397cb0a921f932
2021-05-17T08:12:29.609Z SERIAL « [ACK]                                                                   (0x06)
2021-05-17T08:12:29.614Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-17T08:12:29.615Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:12:29.616Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-17T08:12:29.631Z SERIAL « 0x01070013750000029c                                                 (9 bytes)
2021-05-17T08:12:29.632Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:12:29.632Z DRIVER « [REQ] [SendData]
                                    callback id:     117
                                    transmit status: OK
2021-05-17T08:12:29.753Z SERIAL « 0x0123000400061d9881ab9f3e783fffc2252e84d651af63260299f88f61eebad40 (37 bytes)
                                  7117f0b27
2021-05-17T08:12:29.755Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:12:29.757Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCCommandEncapsulation]
                                    │ sequenced: false
                                    └─[NotificationCCReport]
                                        notification type:   Home Security
                                        notification status: 255
                                        notification state:  Motion detection
2021-05-17T08:12:29.760Z CNTRLR   [Node 006] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 8 => 8
2021-05-17T08:12:34.533Z SERIAL « 0x010c00040006063105030a05e02f                                      (14 bytes)
2021-05-17T08:12:34.535Z CNTRLR   [Node 006] [Multilevel Sensor] Illuminance: metadata updated      [Endpoint 0]
2021-05-17T08:12:34.538Z CNTRLR   [Node 006] [~] [Multilevel Sensor] Illuminance: 1782 => 1504      [Endpoint 0]
2021-05-17T08:12:34.540Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:12:34.542Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[MultilevelSensorCCReport]
                                      type:  Illuminance
                                      scale: Lux
                                      value: 1504
2021-05-17T08:12:39.107Z SERIAL « 0x010c00040006063105030a081bd9                                      (14 bytes)
2021-05-17T08:12:39.109Z CNTRLR   [Node 006] [Multilevel Sensor] Illuminance: metadata updated      [Endpoint 0]
2021-05-17T08:12:39.112Z CNTRLR   [Node 006] [~] [Multilevel Sensor] Illuminance: 1504 => 2075      [Endpoint 0]
2021-05-17T08:12:39.114Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:12:39.117Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[MultilevelSensorCCReport]
                                      type:  Illuminance
                                      scale: Lux
                                      value: 2075
2021-05-17T08:13:10.187Z SERIAL « 0x0108000400060298402f                                              (10 bytes)
2021-05-17T08:13:10.188Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:13:10.189Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCNonceGet]
2021-05-17T08:13:10.195Z SERIAL » 0x01110013060a9880bb6a87fcdc45c71b057675                            (19 bytes)
2021-05-17T08:13:10.196Z DRIVER » [Node 006] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      118
                                  └─[SecurityCCNonceReport]
                                      nonce: 0xbb6a87fcdc45c71b
2021-05-17T08:13:10.199Z SERIAL « [ACK]                                                                   (0x06)
2021-05-17T08:13:10.204Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-17T08:13:10.205Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:13:10.205Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-17T08:13:10.360Z SERIAL « 0x0124000400061e988115751b919e0f7b67b2b78b784d5c12e0da53a4bbc558012 (38 bytes)
                                  c18b2c916ff
2021-05-17T08:13:10.361Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:13:10.364Z DRIVER « [Node 006] [REQ] [ApplicationCommand]
                                  └─[SecurityCCCommandEncapsulation]
                                    │ sequenced: false
                                    └─[NotificationCCReport]
                                        notification type:   7
                                        notification status: 255
                                        notification event:  0x00
                                        event parameters:    
2021-05-17T08:13:10.367Z CNTRLR   [Node 006] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 8 => 0
2021-05-17T08:13:10.551Z SERIAL « 0x0107001376010023bf                                                 (9 bytes)
2021-05-17T08:13:10.552Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:13:10.554Z DRIVER « [REQ] [SendData]
                                    callback id:     118
                                    transmit status: NoAck
2021-05-17T08:13:10.561Z CNTRLR   [Node 006] did not respond after 1/3 attempts. Scheduling next try in 500 ms.
2021-05-17T08:13:11.067Z SERIAL » 0x01110013060a9880bb6a87fcdc45c71b057675                            (19 bytes)
2021-05-17T08:13:11.069Z DRIVER » [Node 006] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      118
                                  └─[SecurityCCNonceReport]
                                      nonce: 0xbb6a87fcdc45c71b
2021-05-17T08:13:11.072Z SERIAL « [ACK]                                                                   (0x06)
2021-05-17T08:13:11.081Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-17T08:13:11.082Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:13:11.082Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-17T08:13:11.178Z SERIAL « 0x010700137601000a96                                                 (9 bytes)
2021-05-17T08:13:11.182Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:13:11.183Z DRIVER « [REQ] [SendData]
                                    callback id:     118
                                    transmit status: NoAck
2021-05-17T08:13:11.187Z CNTRLR   [Node 006] did not respond after 2/3 attempts. Scheduling next try in 500 ms.
2021-05-17T08:13:11.693Z SERIAL » 0x01110013060a9880bb6a87fcdc45c71b057675                            (19 bytes)
2021-05-17T08:13:11.695Z DRIVER » [Node 006] [REQ] [SendData]
                                  │ transmit options: 0x05
                                  │ callback id:      118
                                  └─[SecurityCCNonceReport]
                                      nonce: 0xbb6a87fcdc45c71b
2021-05-17T08:13:11.698Z SERIAL « [ACK]                                                                   (0x06)
2021-05-17T08:13:11.707Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-05-17T08:13:11.708Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:13:11.708Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-05-17T08:13:11.839Z SERIAL « 0x010700137601000e92                                                 (9 bytes)
2021-05-17T08:13:11.840Z SERIAL » [ACK]                                                                   (0x06)
2021-05-17T08:13:11.841Z DRIVER « [REQ] [SendData]
                                    callback id:     118
                                    transmit status: NoAck
2021-05-17T08:13:11.845Z CNTRLR   [Node 006] The node did not respond after 3 attempts.
                                  It is probably asleep, moving its messages to the wakeup queue.
2021-05-17T08:13:11.849Z CNTRLR   [Node 006] The node is now asleep.
f45tb00t commented 3 years ago

Tried it now again. Jumped in front of the sensor (guess when my neighbors see me they think im crazy), but the motion detection was send again. Also the Status went back to OFF now. Just a second ago a tried it again and the status went also back to "off" after the dead time of 30 seconds.

AlCalzone commented 3 years ago

The node seems to be at the edge of connectivity. You can see one nonce report getting sent and received (the node responds with a security encapsulated message), but the acknowledgement for the nonce report does not reach the controller.

This coupled with the secure encapsulation overhead of 2 messages per report is bad news for reliability.

f45tb00t commented 3 years ago

What does this mean? It worked before the update? The sensor is 5m far from the controller. Before I ran it with ozw. Also there were no issues with the motion detectors of fibaro. Additionaly the sensors were added as secure device even before.

So actually I don't get the point what I can do here :(

MartinHjelmare commented 3 years ago

You can try to make the network more robust. There are different ways of doing that. Eg by adding devices that are always on and can relay messages, or by moving devices around. You can also try including the affected devices as non secure and see if that makes a difference.

Closing here now as it's not a problem with the integration.

f45tb00t commented 3 years ago

Why closing? This issue occured after the last update. My setup has not been changed and before it went pretty well without any issues for weeks. So surely it must be related to the new update.

MartinHjelmare commented 3 years ago

As we've tried to explain, it doesn't look like it's related to the code update. Until we have other news there's no point in keeping this open.

AlCalzone commented 3 years ago

There is definitely message loss involved. Might just be some new interference near the controller stick, which could explain the full messages getting through, but not the much shorter ACK.

I don't see anything wrong from the software side, except that the driver tries a bit too hard to send the sleeping node a nonce report.

RF is a bit like voodoo. Maybe the neighbor has a new freezer 🤷🏻‍♂️ If the stick doesn't receive a message, the software on the host cannot do anything about it - update or not.

f45tb00t commented 3 years ago

hi.

the only things which changed are:

  1. 3 new sonoff power plugs. but 2.4 Ghz. So no interference on the zwave band at all. Prooved with my hackRF.
  2. I got a new fridge in the kitchen. Also no interferences at all.

Last but not least we talk about a distance of 5 meters from controller to the device in the kitchen. There is a 15cm wall between. And the fact that it worked before for weeks without any issues, but the update broke it directly after x hours. Also other sensors worked well with a distance more than 20 meters and several walls inbetween. By now the nearest sensor is not working anymore. It makes absolutely no sense, that now 3 sensors are broken at the same time. Also here is no other person on air on that frequency. At least not in a range of 500 meters.

"RF is a bit like voodoo. Maybe the neighbor has a new freezer 🤷🏻‍♂️ If the stick doesn't receive a message, the software on the host cannot do anything about it - update or not." Na, its not really like that. Suprisingly I work in that field. ;) So networks are quite common to me. Also how to tshoot them.

f45tb00t commented 3 years ago

By the way. Since I moved to a new device and still have the old RPi installed(which is not updated to the recent version) I can compare both setups with each other. I'll keep you posted. Both zwavejs and ozw are still configured on the RPi

AlCalzone commented 3 years ago

If you have one, you could try logging with a Zniffer directly next to the stick, to see what radio traffic reaches it.

f45tb00t commented 3 years ago

Okay. I tested it now with my old setup with ozw on the RPi. I have NO issues with ozw and the Fibaro Sensors. In fact the only reason why I migrated from OZW to zwavejs is that the doorlock did not work that well.

How can I downgrade to an older version of zwavejs prior this changes you made which broke my setup? I downgraded HA already to 2021.4.1 but unfortunately zwavejs is still on version v7.4.0... for what ever reason.

And there is also no other device interfering the band of my zwave devices at all. I sniffed the whole night and day. Nothin but my own devices.

AlCalzone commented 3 years ago

Okay then, can you share a zwave-js log where it doesn't work? The log above showed a connectivity issue, but both the "Motion detected" event and the "Idle" event 40 seconds later were received and handled:

2021-05-17T08:12:29.760Z CNTRLR   [Node 006] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 8 => 8
2021-05-17T08:13:10.367Z CNTRLR   [Node 006] [~] [Notification] Home Security[Motion sensor status] [Endpoint 0]
                                  : 8 => 0

If you happen to catch a log where it doesn't return to idle, maybe I can investigate using that.

Again, from the logs above it looks likely that sometimes the idle messages do not reach the stick. The software that's running on the host has zero influence on that - we can't tell the stick to listen differently. And the fact that you saw no interference does not mean that the stick saw a complete message.

f45tb00t commented 3 years ago

@AlCalzone I will see.

But either way what the logs says. It makes absolutely, from a network perspective, very less to no sense at all, that the motion sensor is not going back to idle, but in fact same device, the tampering sensor goes to ALERT when I shake that device, but the tampering (seismic) alert goes back to idle, and the motion sensor stucks meanwhile still at motion detected and stays there.

Surely I could switch back to OZW, because it was rock solid for me, beside that the doorlock worked very shitty. So I was just amazed and happy to use your code as it worked oob with all devices flawelessly.

I will try to get a good log where you can see motion stucks, seismic works. Because if anything stucks, all "should" stuck. It makes not really sense to change the state of one sensor but keep the state of the other at all. Especially since we talking here about bytes. But surely we can even dig more into the deep and analyze every ***** layer, since I really want to have it working again. :)

f45tb00t commented 3 years ago

back. Okay it is definetely something wrong here. Even the debug logs are not trustworthy nor reliable at all.

  1. sensor in bedroom stucked. So I enabled logging due to set it to debug 
    Abonniert Z-Wave JS-Protokollnachrichten ...
2021-05-18T20:46:11.036Z CNTRLR   [Node 007] did not respond after 1/3 attempts. Scheduling next try in 500 ms.
2021-05-18T20:46:11.681Z CNTRLR   [Node 007] did not respond after 2/3 attempts. Scheduling next try in 500 ms.
2021-05-18T20:46:12.361Z CNTRLR   [Node 007] The node did not respond after 3 attempts.
                              It is probably asleep, moving its messages to the wakeup queue.
2021-05-18T20:46:15.917Z CNTRLR   [Node 007] did not respond after 1/3 attempts. Scheduling next try in 500 ms.
2021-05-18T20:46:16.557Z CNTRLR   [Node 007] did not respond after 2/3 attempts. Scheduling next try in 500 ms.
2021-05-18T20:46:17.234Z CNTRLR   [Node 007] The node did not respond after 3 attempts.
                              It is probably asleep, moving its messages to the wakeup queue.
2021-05-18T20:48:48.735Z CNTRLR   [Node 007] did not respond after 1/3 attempts. Scheduling next try in 500 ms.

Thats it. But... meanwhile

Check the Pictures, the values and especially the timestamps of the pictures. The motion sensor status stuck all the time, then I moved to the bedrom, turned on the light and shaked the sensor and put it back to its place. The seismic sensor triggered, the light sensor triggered, the motion sensor stucks all the time, and even the temperature changed.

Really would like to give you more details from debug log... But as you can see its kinda empty.

Bildschirmfoto von 2021-05-18 22-45-22 Bildschirmfoto von 2021-05-18 22-47-51 Bildschirmfoto von 2021-05-18 22-48-51

It would be awesome if we can open this incident again, as I don't see any issues with my setup and network at all. Even during the issues with the sensors for example the doorlock worked as usual. And the doorlock is the one with the highest distance from the controller. Furthermore I sniffed on air with the hackRF. There is nothing interfering on that band when the issues occured again.

I would be pleased to help you to track that issue down. So if I can do anything else, please let me know. Meanwhile can you tell me how I can rollback to an older version please?

AlCalzone commented 3 years ago 
1. So I enabled logging due to set it to debug

That's not debug logging - there should be lines with SERIAL in there.

f45tb00t commented 3 years ago

Bildschirmfoto von 2021-05-19 08-00-28

AlCalzone commented 3 years ago

The HA devs have to chime in then - I only know that was no "debug" level log.

f45tb00t commented 3 years ago

@AlCalzone: I think the screenshots are quite obvious, that something must be wrong with the recent updates. Even if the logging did not work as expected that time.

Can you explain me how it can happen that 3 sensors of the same device act normal and only the motion sensor does not?

--> How can I downgrade?

Thanks for help! :)

AlCalzone commented 3 years ago

--> How can I downgrade?

I don't know - I don't use HA.

Can you explain me how it can happen that 3 sensors of the same device act normal and only the motion sensor does not?

That's what I need the debug logs for :) Any chance you could try the zwavejs2mqtt addon to get to the bottom of this? At least there I could help you a bit more.

f45tb00t commented 3 years ago

Since I'm using the core version of HA in a venv, I had to install a server as well. I don't have any plugins/addos/... .Therefore I've installed zwavejs2mqtt manually already. Because the docu is much better than of zwave server.

https://github.com/zwave-js/zwave-js-server vs https://github.com/zwave-js/zwavejs2mqtt

Just let me know what you need. :)

AlCalzone commented 3 years ago

Just let me know what you need. :)

Try the device with zwavejs2mqtt and see if it "sticks" there as well and make a log of that. I'm relatively sure that setting the loglevel works there and you can download the logfile through the UI.

f45tb00t commented 3 years ago

What do you mean with "try the device with zwavejs2mqtt"? The devices are added there already as this is my server where the integration points to port 3000(TCP). At first I needed to setup zwavejs2mqtt, then I was able to enable the integration and point it to the server for the interaction.

Which Logs do you want? There are at least two options to enable logs

  1. Settings - General - (Log enabled, Log to File, Debbuging)
  2. Settings - Zwave - (Log enabled, Log to File, Debbuging) => All enabled and set to debugging
AlCalzone commented 3 years ago

What do you mean with "try the device with zwavejs2mqtt"?

I meant that you should reproduce the problem there. I need the log from the 2nd logger (Zwave)

f45tb00t commented 3 years ago

Roger that.

OMG... That will be a lots of logs :-D I'll just let in run and check HA if it stucks again. Or I'll make it even more easy. I'll create an automation that will send me a message when the status stucks again

I'll keep you posted

AlCalzone commented 3 years ago

That will be a lots of logs

That's fine. As long as I know approximately when it happened.

f45tb00t commented 3 years ago

Okay got it. But just the other way around. Motion stucked, then I moved to the kitchen, shaked the sensor and the seismisc senser got "ON". Now the motion sensor is back on idle, but the seismic sensor stucks. Its node 6. But even without shaking, the motion sensor went 0 -> 8 and 8 -> 0 again, meanwhile the seismic sensor still stuck. 

2021-05-19 16:41:38.390 INFO ZWAVE: Node 6: value updated: 113-0-Home Security-Motion sensor status 0 => 8
2021-05-19 16:58:20.309 INFO ZWAVE: Node 6: value updated: 113-0-Home Security-Motion sensor status 8 => 8
2021-05-19 16:58:22.171 INFO ZWAVE: Node 6: value updated: 113-0-Home Security-Cover status 0 => 3
2021-05-19 16:58:51.264 INFO ZWAVE: Node 6: value updated: 113-0-Home Security-Motion sensor status 8 => 0
2021-05-19 17:00:36.457 INFO ZWAVE: Node 6: value updated: 113-0-Home Security-Motion sensor status 0 => 8
2021-05-19 17:01:07.381 INFO ZWAVE: Node 6: value updated: 113-0-Home Security-Motion sensor status 8 => 0

zwavejs_1.log_2021-05-19.log.txt zwavejs2mqtt.log.txt