search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
72.64k stars 30.4k forks source link

All my Z-wave devices become dead after a few hours #92521

Closed GAguado closed 11 months ago

GAguado commented 1 year ago

The problem

Dear Team, After a few years without any relevant issues, I am suffering a rare behavior with Z-Wave-JS integration, every few hours all z-wave devices become dead and can not be controlled. I did not change any device, about 73, not any signal-related problems, and no delays. After HA restart (one or more times) it becomes fully operational. Running on RPi 4b, HAS 2023.4.6 Z-Wave JS UI 1.11.2 Z-Wave JS Driver 10.16.0 - 1.28.0

Z-Wave JS Log: anything, not any single line (Any device) Log: Z-Wave error 202: Failed to send the command after 3 attempts (Status fail) ZW0202 (ERROR) Home Assistant WebSocket API

When this occurs the rest of HA keeps running Any hint? Best Regards Gerardo

What version of Home Assistant Core has the issue?

core-2023.4.6

What was the last working version of Home Assistant Core?

core-2023.3.x

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Z-Wave JS UI

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 year ago

Hey there @home-assistant/z-wave, mind taking a look at this issue as it has been labeled with an integration (zwave_js) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `zwave_js` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign zwave_js` Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)

zwave_js documentation zwave_js source (message by IssueLinks)

turbo4door commented 1 year ago

I'm also experiencing this issue with my thermostat. It started yesterday, no change to device or HA that I'm aware of. Device just becomes unavailable. Pinging from the device info page or pressing a button on the thermostat brings it back for a while but it eventually becomes unavailable again. Only seems to be this device, which is mains powered. Battery powered smoke/co2 detectors are not having this issue. Tried updating to latest version of Zwave js but no luck so far.

Driver version 10.17.0 Server version 1.28.0 HA 2023.4.6

villar3 commented 1 year ago

hi,

i confirm a similar issue on my device. I need to restart all my HA (restart system) to solve temporary this issue. i think this bug was introducing there are 1 month. thanks for your help.

stefanes commented 1 year ago

I have experienced this issue, or something similar (or maybe two different issues). For me the issue seems to appear every few days, and it manifests itself in two different ways:

  1. Sometimes all devices turns up dead as described above
  2. Sometimes devices just "get stuck" either on or off

When 1) happens, attempting to turn the device on/off generates an API call error. When 2) happens, if turning a device on/off absolutely nothing happens. So far I have not found anything in any logs.

In both instances the issue seems to go away by restarting the Z-Wave JS UI add-on. I unfortunately have no good backup with a previous version of the add-on since I made quite a few changes at the same time as I updated the Z-Wave JS UI add-on. If there is any other way of downgrading (temporarily for test) I would be more than happy do that?

Component Version
Home Assistant 2023.5.2
Supervisor 2023.04.1
Operating System 10.1
Frontend 20230503.3 - latest
Z-Wave JS Driver 10.17.1
Z-Wave JS Server 1.28.0
Z-Wave JS UI add-on 1.12.0
Aeotec Z-Stick 7 7.19.2
RF Region EU
GAguado commented 1 year ago

Months ago and from time to time I got one or two devices dead that could be revived with just a ping BUT what is happening now is that ALL Zwave devices become dead and the only way to get them alive is restarting the Z-Wave JS UI add-on. In my case, I don't see any problem with time-outs in my network, is the same network I had years ago. I am using a 700 series controller but it was the same months ago and I didn't suffer such an effect.

MartinHjelmare commented 1 year ago

Please try to capture driver debug logs of the time that causes the device(s) to go dead.

stefanes commented 1 year ago

Please try to capture driver debug logs of the time that causes the device(s) to go dead.

Just to be clear, by driver debug logs you mean from under the Logs tab here, with Log Level set to Debug?

image

Or do you mean this?

image

MartinHjelmare commented 1 year ago

The former is the driver logs. The latter will set debug level for the integration logs. Both may be interesting but driver logs is the minimum.

Start with the driver logs at debug level as you've written.

stefanes commented 1 year ago

Had this happen early this morning and got some logs, not sure if they can contain sensitive information or are safe to attach?

I now believe the issue actually manifests itself in the same way every time, not in two different ways as I claim above:

  1. When the issue happens, all Z-Wave devices becomes unresponsive or "get stuck" (see in 2) above)
  2. After a while the devices are finally marked as unavailable in Home Assistant (see 1) above)

I can see this sequence in my logs as I created a few automations to detect the issue and automatically restart Z-Wave JS UI. I got it wrong though, so instead of restarting the addon after detecting devices being "stuck" I get a warning in the logs instead of actually restarting the addon. Later, another automation detects a high number of unavailable devices and successfully restarts the addon... 🙂

2023-05-18 06:49:39.028 ERROR (MainThread) [homeassistant.components.script.turn_off_zwave_switch_group] Turn off switch group (Z-Wave multicast): Error executing script. Error for call_service at pos 2: Unable to set value via multicast
...
2023-05-18 06:59:28.808 ERROR (MainThread) [homeassistant.components.script.turn_off_zwave_switch_group] Turn off switch group (Z-Wave multicast): Error executing script. Error for call_service at pos 2: Unable to set value via multicast
2023-05-18 06:59:38.940 WARNING (MainThread) [homeassistant.helpers.service] Unable to find referenced entities automation.z_wave_addon_restart or it is/they are currently not available
2023-05-18 06:59:39.446 ERROR (MainThread) [homeassistant.components.script.turn_off_zwave_switch_group] Turn off switch group (Z-Wave multicast): Error executing script. Error for call_service at pos 2: Unable to set value via multicast
...
2023-05-18 07:39:36.907 ERROR (MainThread) [homeassistant.components.script.turn_off_zwave_switch_group] Turn off switch group (Z-Wave multicast): Error executing script. Error for call_service at pos 2: Unable to set value via multicast
2023-05-18 07:39:43.879 WARNING (MainThread) [homeassistant.config_entries] Config entry 'Z-Wave JS' for zwave_js integration not ready yet: Failed to connect: Cannot connect to host a0d7b954-zwavejs2mqtt:3000 ssl:default [Connect call failed ('172.30.33.1', 3000)]; Retrying in background
...

The driver log contains a bunch of ZW0202 and other failures, the add-on is then restarted at around 07:39 CEST (05:39 UTC):

...
2023-05-18T05:00:31.940Z CNTRLR   [Node 009] The node did not respond after 1 attempts, it is presumed dead
2023-05-18T05:00:31.984Z CNTRLR   [Node 009] ping failed: Failed to send the command after 1 attempts (Status Fa
                                  il) (ZW0202)
2023-05-18T05:00:31.996Z SERIAL » 0x010d00a9012301002500000000b2ef                                    (15 bytes)
2023-05-18T05:00:32.001Z DRIVER » [Node 035] [REQ] [SendDataBridge]
                                  │ source node id:   1
                                  │ transmit options: 0x25
                                  │ callback id:      178
                                  └─[NoOperationCC]
2023-05-18T05:00:32.010Z SERIAL « [ACK]                                                                   (0x06)
2023-05-18T05:00:32.029Z SERIAL « 0x010401a90152                                                       (6 bytes)
2023-05-18T05:00:32.031Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T05:00:32.034Z DRIVER « [RES] [SendDataBridge]
                                    was sent: true
2023-05-18T05:00:32.043Z SERIAL « 0x011d00a9b202000000000000000000000000000000000000000000000000fb    (31 bytes)
2023-05-18T05:00:32.045Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T05:00:32.048Z DRIVER « [REQ] [SendDataBridge]
                                    callback id:                           178
                                    transmit status:                       Fail, took 0 ms
                                    routing attempts:                      0
                                    protocol & route speed:                Unknown (0x00)
                                    ACK RSSI:                              0 dBm
                                    ACK channel no.:                       0
                                    TX channel no.:                        0
                                    TX power:                              0 dBm
                                    measured noise floor:                  0 dBm
                                    ACK TX power by destination:           0 dBm
                                    measured RSSI of ACK from destination: 0 dBm
                                    measured noise floor by destination:   0 dBm
...
2023-05-18T05:35:22.913Z CNTRLR   [Node 034] The node did not respond after 1 attempts, it is presumed dead
2023-05-18T05:35:22.927Z CNTRLR   [Node 034] ping failed: Failed to send the command after 1 attempts (Status Fa
                                  il) (ZW0202)
2023-05-18T05:35:28.208Z SERIAL » 0x010e00ab0103180a0c03250100258bcf                                  (16 bytes)
2023-05-18T05:35:28.210Z DRIVER » [REQ] [SendDataMulticastBridge]
                                  │ source node id:   1
                                  │ target nodes:     24, 10, 12
                                  │ transmit options: 0x25
                                  │ callback id:      139
                                  └─[BinarySwitchCCSet]
                                      target value: false
2023-05-18T05:35:28.234Z SERIAL « [ACK]                                                                   (0x06)
2023-05-18T05:35:28.239Z SERIAL « 0x010401ab0150                                                       (6 bytes)
2023-05-18T05:35:28.255Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T05:35:28.347Z DRIVER « [RES] [SendDataMulticastBridge]
                                    was sent: true
2023-05-18T05:35:28.369Z SERIAL « 0x010500ab8b02d8                                                     (7 bytes)
2023-05-18T05:35:28.372Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T05:35:28.375Z DRIVER « [REQ] [SendDataMulticastBridge]
                                    callback id:     139
                                    transmit status: Fail
2023-05-18T05:35:38.703Z SERIAL » 0x010e00ab0103180a0c03250100258cc8                                  (16 bytes)
2023-05-18T05:35:38.712Z DRIVER » [REQ] [SendDataMulticastBridge]
                                  │ source node id:   1
                                  │ target nodes:     24, 10, 12
                                  │ transmit options: 0x25
                                  │ callback id:      140
                                  └─[BinarySwitchCCSet]
                                      target value: false
2023-05-18T05:35:38.740Z SERIAL « [ACK]                                                                   (0x06)
2023-05-18T05:35:38.752Z SERIAL « 0x010401ab0150                                                       (6 bytes)
2023-05-18T05:35:38.765Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T05:35:38.877Z DRIVER « [RES] [SendDataMulticastBridge]
                                    was sent: true
2023-05-18T05:35:38.890Z SERIAL « 0x010500ab8c02df                                                     (7 bytes)
2023-05-18T05:35:38.895Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T05:35:38.909Z DRIVER « [REQ] [SendDataMulticastBridge]
                                    callback id:     140
                                    transmit status: Fail
2023-05-18T05:35:43.964Z SERIAL » 0x0103003bc7                                                         (5 bytes)
2023-05-18T05:35:43.967Z DRIVER » [REQ] [GetBackgroundRSSI]
2023-05-18T05:35:43.982Z SERIAL « [ACK]                                                                   (0x06)
2023-05-18T05:35:43.992Z SERIAL « 0x0107013b9093937f2d                                                 (9 bytes)
2023-05-18T05:35:44.014Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T05:35:44.092Z DRIVER « [RES] [GetBackgroundRSSI]
                                    channel 0: -112 dBm
                                    channel 1: -109 dBm
                                    channel 2: -109 dBm
MartinHjelmare commented 1 year ago

@AlCalzone any idea for further troubleshooting?

AlCalzone commented 1 year ago

I usually see transmit status: Fail in two situations:

  1. The driver tries to send a frame that is too large - but this should no longer happen since a few versions ago - and a ping is definitely not too large
  2. The controller is not able to transmit anything because the network is flooded. I have an open issue for this so the nodes don't get marked as dead, but rather the controller should be marked as jammed or something like that: https://github.com/zwave-js/node-zwave-js/issues/4191

The logs above don't contain enough context from before this happens, but my guess is 2, especially since it happens multiple times in a row (and for a ping).

stefanes commented 1 year ago

I have a log from long before the issue happens that I can attach, if they contain no sensitive information?

This issues is new since updating the add-on, never seen it before and I have not added any new devices to the network for quite some time.

AlCalzone commented 1 year ago

Unless you're actively changing some user codes, the logs contain nothing sensitive. Although I think even those are obfuscated now. The last 1-2 minutes should be enough for this though.

stefanes commented 1 year ago

zwave_js.log

AlCalzone commented 1 year ago

Ok this one is weird.

It exhibits the typical symptoms of too much reporting combined with bad connectivity, for example in the 30 minutes leading up to the first time the controller fails to send, I count 987 meter reports, 148 multilevel sensor reports, which averages to 0.6 reports/s. This sounds manageable, but most of this happens in bursts where you're looking at in the 8-10 reports/s range.

For example, here's a stretch where Node 22 tries to reach the controller and seems to get no acknowledgement, which is why it eventually switches to broadcasting:

2023-05-18T04:37:22.226Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:22.241Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:22.270Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:22.290Z SERIAL « 0x011300a80001160a3202214400001293000000a528                        (21 bytes)
2023-05-18T04:37:22.297Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:22.310Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:22.325Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:22.349Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:22.370Z SERIAL « 0x011300a80001160a3202214400001293000000a528                        (21 bytes)
2023-05-18T04:37:22.372Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:22.384Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:22.403Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:22.409Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:22.431Z SERIAL « 0x011300a80001160a3202214400001293000000a528                        (21 bytes)
2023-05-18T04:37:22.435Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:22.445Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:22.458Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:22.470Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:22.486Z SERIAL « 0x011300a80001160a3202214400001293000000a528                        (21 bytes)
2023-05-18T04:37:22.490Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:22.497Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:22.515Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:22.529Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:22.546Z SERIAL « 0x011300a80001160a3202214400001293000000a528                        (21 bytes)
2023-05-18T04:37:22.551Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:22.567Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:22.579Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:22.590Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:22.691Z SERIAL « 0x011300a80001160a3202214400001293000000a528                        (21 bytes)
2023-05-18T04:37:22.698Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:22.705Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:22.726Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:22.740Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:22.866Z SERIAL « 0x011300a80001160a3202214400001293000000a528                        (21 bytes)
2023-05-18T04:37:22.870Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:22.876Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:22.887Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:22.900Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:23.017Z SERIAL « 0x011300a80001160a3202214400001293000000a12c                        (21 bytes)
2023-05-18T04:37:23.021Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:23.028Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:23.039Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:23.053Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -95 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:23.756Z SERIAL « 0x011300a80001160a3202214400001293000000a32e                        (21 bytes)
2023-05-18T04:37:23.763Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:23.772Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:23.783Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:23.800Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -93 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:23.821Z SERIAL « 0x011300a80001160a3202214400001293000000a32e                        (21 bytes)
2023-05-18T04:37:23.826Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:23.838Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:23.850Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:23.894Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -93 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:23.927Z SERIAL « 0x011300a80001160a3202214400001293000000a32e                        (21 bytes)
2023-05-18T04:37:23.933Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:23.943Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:23.971Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:23.982Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -93 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:24.250Z SERIAL « 0x011300a80001160a3202214400001293000000a62b                        (21 bytes)
2023-05-18T04:37:24.255Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:24.260Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:24.289Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:24.295Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -90 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:24.333Z SERIAL « 0x011300a80001160a3202214400001293000000a62b                        (21 bytes)
2023-05-18T04:37:24.336Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:24.365Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:24.371Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:24.377Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -90 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:24.396Z SERIAL « 0x011300a80001160a3202214400001293000000a62b                        (21 bytes)
2023-05-18T04:37:24.398Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:24.402Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:24.415Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:24.421Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -90 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:24.724Z SERIAL « 0x011300a80001160a32022144000012930000009815                        (21 bytes)
2023-05-18T04:37:24.729Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:24.734Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:24.744Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:24.761Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -104 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:24.905Z SERIAL « 0x011300a81401160a3202214400001293000000a53c                        (21 bytes)
2023-05-18T04:37:24.909Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:24.914Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:24.924Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:24.937Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:25.019Z SERIAL « 0x011300a81401160a32022144000012930000009f06                        (21 bytes)
2023-05-18T04:37:25.030Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:25.050Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:25.068Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:25.077Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -97 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:25.088Z SERIAL « 0x011300a81401160a3202214400001293000000a930                        (21 bytes)
2023-05-18T04:37:25.093Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:25.098Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:25.122Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:25.142Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -87 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:25.214Z SERIAL « 0x011300a81401160a3202214400001293000000a33a                        (21 bytes)
2023-05-18T04:37:25.218Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:25.231Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:25.251Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:25.261Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -93 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:25.272Z SERIAL « 0x011300a81401160a3202214400001293000000a138                        (21 bytes)
2023-05-18T04:37:25.275Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:25.285Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:25.297Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:25.306Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -95 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds
2023-05-18T04:37:25.321Z SERIAL « 0x011300a81401160a32022144000012930000009f06                        (21 bytes)
2023-05-18T04:37:25.324Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:25.328Z CNTRLR   [Node 022] [Meter] value[65537]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:25.350Z CNTRLR   [Node 022] [~] [Meter] value[65537]: 47.55 => 47.55               [Endpoint 0]
2023-05-18T04:37:25.363Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -97 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      kWh
                                      rate type:  Consumed
                                      value:      47.55
                                      time delta: 0 seconds

that's 22 messages in roughly 3 seconds, then it seems to give up on sending this measurement and switches to the power consumption:

2023-05-18T04:37:28.875Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:28.906Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:28.912Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:28.922Z SERIAL « 0x011100a800011608320221120004000000a5fb                            (19 bytes)
2023-05-18T04:37:28.924Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:28.929Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:28.945Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:28.953Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:28.966Z SERIAL « 0x011100a800011608320221120004000000a5fb                            (19 bytes)
2023-05-18T04:37:28.970Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:28.975Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:28.987Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:28.995Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:29.015Z SERIAL « 0x011100a800011608320221120004000000a5fb                            (19 bytes)
2023-05-18T04:37:29.018Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:29.022Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:29.047Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:29.054Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:29.068Z SERIAL « 0x011100a800011608320221120004000000a5fb                            (19 bytes)
2023-05-18T04:37:29.071Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:29.080Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:29.095Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:29.114Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:29.559Z SERIAL « 0x011100a800011608320221120004000000a5fb                            (19 bytes)
2023-05-18T04:37:29.570Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:29.577Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:29.587Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:29.594Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:29.755Z SERIAL « 0x011100a800011608320221120004000000a4fa                            (19 bytes)
2023-05-18T04:37:29.758Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:29.763Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:29.780Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:29.787Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -92 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:29.821Z SERIAL « 0x011100a8000116083202211200040000009cc2                            (19 bytes)
2023-05-18T04:37:29.831Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:29.842Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:29.858Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:29.871Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -100 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:29.898Z SERIAL « 0x011100a800011608320221120004000000a5fb                            (19 bytes)
2023-05-18T04:37:29.901Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:29.906Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:29.918Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:29.932Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:29.950Z SERIAL « 0x011100a8000116083202211200040000009dc3                            (19 bytes)
2023-05-18T04:37:29.957Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:29.966Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:29.974Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:29.977Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -99 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:30.012Z SERIAL « 0x011100a8000116083202211200040000009cc2                            (19 bytes)
2023-05-18T04:37:30.016Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:30.027Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:30.038Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:30.048Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -100 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:30.103Z SERIAL « 0x011100a8000116083202211200040000009bc5                            (19 bytes)
2023-05-18T04:37:30.106Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:30.114Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:30.130Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:30.139Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -101 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:30.521Z SERIAL « 0x011100a800011608320221120004000000a3fd                            (19 bytes)
2023-05-18T04:37:30.529Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:30.534Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:30.542Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:30.557Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -93 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:30.583Z SERIAL « 0x011100a800011608320221120004000000a3fd                            (19 bytes)
2023-05-18T04:37:30.586Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:30.599Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:30.608Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:30.627Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -93 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:30.708Z SERIAL « 0x011100a800011608320221120004000000a3fd                            (19 bytes)
2023-05-18T04:37:30.712Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:30.724Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:30.740Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:30.748Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -93 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:30.769Z SERIAL « 0x011100a800011608320221120004000000a3fd                            (19 bytes)
2023-05-18T04:37:30.774Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:30.789Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:30.797Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:30.828Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -93 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:31.124Z SERIAL « 0x011100a800011608320221120004000000a5fb                            (19 bytes)
2023-05-18T04:37:31.135Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:31.140Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:31.163Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:31.172Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:31.189Z SERIAL « 0x011100a800011608320221120004000000a5fb                            (19 bytes)
2023-05-18T04:37:31.192Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:31.198Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:31.208Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:31.220Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:31.244Z SERIAL « 0x011100a800011608320221120004000000a5fb                            (19 bytes)
2023-05-18T04:37:31.248Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:31.258Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:31.277Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:31.290Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ RSSI: -91 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:31.318Z SERIAL « 0x011100a8140116083202211200040000009fd5                            (19 bytes)
2023-05-18T04:37:31.322Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:31.336Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:31.358Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:31.366Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -97 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:31.396Z SERIAL « 0x011100a814011608320221120004000000a3e9                            (19 bytes)
2023-05-18T04:37:31.402Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:31.432Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:31.445Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:31.464Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -93 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:31.484Z SERIAL « 0x011100a8140116083202211200040000009fd5                            (19 bytes)
2023-05-18T04:37:31.490Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:31.500Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:31.523Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:31.534Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -97 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:31.562Z SERIAL « 0x011100a814011608320221120004000000a4ee                            (19 bytes)
2023-05-18T04:37:31.569Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:31.578Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:31.588Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:31.598Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -92 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:31.618Z SERIAL « 0x011100a8140116083202211200040000009ad0                            (19 bytes)
2023-05-18T04:37:31.621Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:31.628Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:31.644Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:31.649Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -102 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:31.676Z SERIAL « 0x011100a814011608320221120004000000a0ea                            (19 bytes)
2023-05-18T04:37:31.680Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:31.686Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:31.696Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:31.713Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -96 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds
2023-05-18T04:37:31.736Z SERIAL « 0x011100a814011608320221120004000000a9e3                            (19 bytes)
2023-05-18T04:37:31.740Z SERIAL » [ACK]                                                                   (0x06)
2023-05-18T04:37:31.747Z CNTRLR   [Node 022] [Meter] value[66049]: metadata updated                 [Endpoint 0]
2023-05-18T04:37:31.755Z CNTRLR   [Node 022] [~] [Meter] value[66049]: 4 => 4                       [Endpoint 0]
2023-05-18T04:37:31.763Z DRIVER « [Node 022] [REQ] [BridgeApplicationCommand]
                                  │ type:        broadcast
                                  │ target node: 255
                                  │ RSSI:        -87 dBm
                                  └─[MeterCCReport]
                                      type:       Electric
                                      scale:      W
                                      rate type:  Consumed
                                      value:      4
                                      time delta: 0 seconds

which it attempts 26 times, then switches to V a couple of seconds later (I'm not going to quote this again). Meanwhile Node 42 also joins in.

Both have very low RSSI values around -90 to -100 dBm, the latter is very close to indistinguishable from the background noise:

2023-05-18T04:47:57.301Z DRIVER « [RES] [GetBackgroundRSSI]
                                    channel 0: -112 dBm
                                    channel 1: -109 dBm
                                    channel 2: -109 dBm

The insane repetition here is most likely because the nodes don't hear that the controller received the message and retry over and over. You will need to look into improving their reception - more on that later.

This traffic alone is enough to jam a controller. The weird thing is that you end up with radio silence for 12 minutes, and then the controller stops transmitting:

2023-05-18T04:48:39.895Z CNTRLR   [Node 022] [Meter] value[67073]: metadata updated                 [Endpoint 0]

... silence ...

2023-05-18T05:00:31.763Z CNTRLR » [Node 009] pinging the node...
...
2023-05-18T05:00:31.921Z DRIVER « [REQ] [SendDataBridge]
                                    callback id:                           177
                                    transmit status:                       Fail, took 0 ms

Maybe it got overloaded by the traffic so much that it stopped doing anything for 12 minutes.

As for troubleshooting, let's make sure the obvious recommendations are followed first:

  1. If this is a 700 series controller, make sure it is on firmware 7.17.2 (lower is broken, higher may be okay, but I cannot vouch for those versions)
  2. If this is an Aeotec Z-Stick 7 (you're in EU, right?), try a different stick. You can migrate an existing network using NVM backup and restore (Z-Wave JS UI only)
  3. Controller placement: https://zwave-js.github.io/node-zwave-js/#/troubleshooting/connectivity-issues?id=general-troubleshooting
  4. Tune your nodes' reporting configuration: https://zwave-js.github.io/node-zwave-js/#/troubleshooting/network-health?id=optimizing-the-reporting-configuration

To improve the connectivity of the aforementioned devices, you can use Z-Wave JS UI: https://zwave-js.github.io/node-zwave-js/#/troubleshooting/network-health?id=testing-the-connection-strength

Here's an issue going through the process where a user had similar problems: https://github.com/zwave-js/node-zwave-js/issues/5766

stefanes commented 1 year ago

This indeed a version 700 controller located in EU - an Aeotec Z-Stick 7, and I am on the latest version 7.19.2.

Interestingly enough both node 22 and 42 are Popp & Co - Popp Smart Outdoor Plug - IP44 rated nodes. Unfortunately I can't seem to tweak what or when they send metrics.

Maybe the make is not relevant here though, both of theses are placed outdoors and I live in an old house that is basically a concrete bunker... They seems to work mostly fine though.

The fact that this started with the version of Z-Wave JS UI I mention above, any idea why this has become an issue only now?

AlCalzone commented 1 year ago

any idea why this has become an issue only now?

Something in or around your house must have changed. With these connectivity issues, the problem is entirely between the stick and the nodes, and Z-Wave JS has the role of a bystander.

They seems to work mostly fine though.

Yeah, but they make the rest of your network not work fine. You can try power-cycling them, it's not unheard of that this sometimes fixes issues with end devices, and at the very least you should get some temporary radio silence.

GAguado commented 1 year ago

Dear Team, Let me recover the issue that I opened three weeks ago and some others used with their related experiences. As I stated in the first message, my case is different. Since a couple of months, more or less, I lose control of ALL Zwave devices. In the past from time to time some devices suddenly appeared dead but most of the time they could be recovered with a simple ping. (With my previous commercial controller I had no such problem) My setup is large but quite stable, untouched for years except that one year ago I moved to HA and also a new controller series 700 (Razberry). 90% of devices have a direct connection with the controller and the rest have only one hope. Background signal level around -110 dBm and devices around 60-70 dBm, latencies in the range 30-100 ms as average. When I lose control all ZWave devices become unresponsive but the rest of HA remains working and this happens every 6-20 hours. I spent a lot of time trying to get the Z-Wave-JS logs just before this happened but I could not notice anything. Please find three different time lons attached, all when the devices are out of control. Thanks for your great work log0.txt log1.txt log2.txt

AlCalzone commented 1 year ago

@GAguado I can't make out any instance where nodes go dead in your logs. log1.txt contains 2 instances where Node 66 is spamming notifications, delaying the controller for about 2 seconds. log2.txt is the same as log1.txt

GAguado commented 1 year ago

You are right, logs are after the crash, normally node 66 is fine, but when Z-Wave-JS crash I noticed any kind of node behavior, look at this node right now: image

WWillig commented 1 year ago

Hi, I am having similar problems. It began with the upgrade from HA 2023.2.x to 2023.3...

I have subsequently upgraded to 2023.5.2 and ZWave-JS to 0.1.83. After about a day the ZWave devices stop working.
I have reloaded 2023.2.x and the devices work properly again. When on 2023.5.2 Restarting does not help but a HA Reboot or ZWave JS restart brings the devices back temporarily.

AlCalzone commented 1 year ago

The next release of node-zwave-js will include a new status property for the controller, which will indicate whether it is currently unable to transmit due to being jammed: https://github.com/zwave-js/node-zwave-js/pull/6174

issue-triage-workflows[bot] commented 11 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.