ZHA Diagnostics suddenly start failing w/ JSON Serialization Error

The problem

Seems a repeat of #106564.

Every so often, the system gets into a state where ZHA diagnostics fail to work. I have an automation that loads the diagnostics every 5m or so, and I sometimes see this for hours:

2024-07-29 19:43:28.994 ERROR (MainThread) [homeassistant.components.diagnostics] Failed to serialize to JSON: config_entry/bd03ab5d4454dbfe36c921a32a8c5a21. Bad data at $.data.application_state.network_info.nwk_addresses<key: 00:15:8d:00:07:c8:d8:73>=00:15:8d:00:07:c8:d8:73(<class 'zigpy.types.named.EUI64'>, $.data.application_state.network_info.nwk_addresses<key: 00:15:8d:00:09:ef:8b:08>=00:15:8d:00:09:ef:8b:08(<class 'zigpy.types.named.EUI64'>, $.data.application_state.network_info.nwk_addresses<key: 00:15:8d:00:0a:d8:d4:56>=00:15:8d:00:0a:d8:d4:56(<class 'zigpy.types.named.EUI64'>, $.data.application_state.network_info.nwk_addresses<key: 00:15:8d:00:0a:d8:f0:74>=00:15:8d:00:0a:d8:f0:74(<class 'zigpy.types.named.EUI64'>, $.data.application_state.counters.controller_app_counters.unicast_rx=unicast_rx = 24(<class 'zigpy.state.Counter'>, $.data.application_state.counters.controller_app_counters.unicast_tx_success=unicast_tx_success = 15(<class 'zigpy.state.Counter'>, $.data.application_state.counters.controller_app_counters.multicast_tx_success_unexpected=multicast_tx_success_unexpected = 1(<class 'zigpy.state.Counter'>, $.data.application_state.counters.controller_app_counters.unicast_tx_failure=unicast_tx_failure = 2(<class 'zigpy.state.Counter'>

What version of Home Assistant Core has the issue?

core-2024.7.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

ZHA

Link to integration documentation on our website

https://www.home-assistant.io/integrations/zha/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-07-29 19:43:28.994 ERROR (MainThread) [homeassistant.components.diagnostics] Failed to serialize to JSON: config_entry/bd03ab5d4454dbfe36c921a32a8c5a21. Bad data at $.data.application_state.network_info.nwk_addresses<key: 00:15:8d:00:07:c8:d8:73>=00:15:8d:00:07:c8:d8:73(<class 'zigpy.types.named.EUI64'>, $.data.application_state.network_info.nwk_addresses<key: 00:15:8d:00:09:ef:8b:08>=00:15:8d:00:09:ef:8b:08(<class 'zigpy.types.named.EUI64'>, $.data.application_state.network_info.nwk_addresses<key: 00:15:8d:00:0a:d8:d4:56>=00:15:8d:00:0a:d8:d4:56(<class 'zigpy.types.named.EUI64'>, $.data.application_state.network_info.nwk_addresses<key: 00:15:8d:00:0a:d8:f0:74>=00:15:8d:00:0a:d8:f0:74(<class 'zigpy.types.named.EUI64'>, $.data.application_state.counters.controller_app_counters.unicast_rx=unicast_rx = 24(<class 'zigpy.state.Counter'>, $.data.application_state.counters.controller_app_counters.unicast_tx_success=unicast_tx_success = 15(<class 'zigpy.state.Counter'>, $.data.application_state.counters.controller_app_counters.multicast_tx_success_unexpected=multicast_tx_success_unexpected = 1(<class 'zigpy.state.Counter'>, $.data.application_state.counters.controller_app_counters.unicast_tx_failure=unicast_tx_failure = 2(<class 'zigpy.state.Counter'>

Additional information

No response

Hey there @dmulcahey, @adminiuga, @puddly, @thejulianjes, mind taking a look at this issue as it has been labeled with an integration (zha) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `zha` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign zha` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

zha documentation zha source _{^{(message by IssueLinks)}}

Note that reloading the integration seems to fix this. I may setup a job to run that on a daily basis. :(

I have an automation that loads the diagnostics every 5m or so

What data are you using? There are disabled entities on the coordinator that encompass all of the counters. Loading diagnostics performs an energy scan of all 16 channels, which effectively disables your Zigbee radio for the duration of the download (5+ seconds).

I have it doing this...can't remember where I got this from....

 - platform: rest
    resource: http://192.168.1.REDACTED:8123/api/diagnostics/config_entry/REDACTED
    name: zha_diagnostics
    icon: "mdi:zigbee"
    value_template: 'OK'
    json_attributes_path: "$.data"
    json_attributes:
      - "energy_scan"
      - "application_state"
    method: GET
    scan_interval: 300
    headers:
      Authorization: !secret haRestApiLongLivedToken
      Content-Type: application/json
  - platform: template
    sensors:
      zha_channel:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'application_state')['network_info']['channel'] }}"
        friendly_name: ZHA Channel
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_11_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['11'] | round(0) }}"
        friendly_name: ZHA Channel 11 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_12_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['12'] | round(0) }}"
        friendly_name: ZHA Channel 12 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_13_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['13'] | round(0) }}"
        friendly_name: ZHA Channel 13 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_14_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['14'] | round(0) }}"
        friendly_name: ZHA Channel 14 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_15_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['15'] | round(0) }}"
        friendly_name: ZHA Channel 15 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_16_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['16'] | round(0) }}"
        friendly_name: ZHA Channel 16 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_17_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['17'] | round(0) }}"
        friendly_name: ZHA Channel 17 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_18_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['18'] | round(0) }}"
        friendly_name: ZHA Channel 18 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_19_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['19'] | round(0) }}"
        friendly_name: ZHA Channel 19 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_20_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['20'] | round(0) }}"
        friendly_name: ZHA Channel 20 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_21_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['21'] | round(0) }}"
        friendly_name: ZHA Channel 21 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_22_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['22'] | round(0) }}"
        friendly_name: ZHA Channel 22 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_23_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['23'] | round(0) }}"
        friendly_name: ZHA Channel 23 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_24_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['24'] | round(0) }}"
        friendly_name: ZHA Channel 24 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_25_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['25'] | round(0) }}"
        friendly_name: ZHA Channel 25 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"
      zha_channel_26_utilization:
        value_template: "{{ state_attr('sensor.zha_diagnostics', 'energy_scan')['26'] | round(0) }}"
        friendly_name: ZHA Channel 26 Utilization
        unit_of_measurement: '%'
        icon_template: "{{'mdi:zigbee'}}"

@puddly - Here's where I got the above idea from: https://github.com/home-assistant/core/issues/91160#issuecomment-1529168994

home-assistant / core