2024.8.2 ZHA fails to start up with `extra keys not allowed @ data['disable_new_entities']`

[Mark612]

The problem

Upgraded from 7.2 to 8.2, and ZHA fails all devices. Tried to reload, same error. Went back to 7.2 and all working fine.

What version of Home Assistant Core has the issue?

core-2024.8.2

What was the last working version of Home Assistant Core?

core-2024.7.2

What type of installation are you running?

Home Assistant Container

Integration causing the issue

ZHA

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-08-18 06:59:15.597 ERROR (DbWorker_0) [homeassistant.components.logbook.processor] Error with zha describe event for zha_event
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/logbook/processor.py", line 247, in _humanify
    data = describe_event(event_cache.get(row))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/logbook.py", line 44, in async_describe_zha_event
    zha_device = async_get_zha_device_proxy(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1014, in async_get_zha_device_proxy
    zha_gateway_proxy = get_zha_gateway_proxy(hass)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 993, in get_zha_gateway_proxy
    raise ValueError("No gateway object exists")
ValueError: No gateway object exists
2024-08-18 06:59:15.598 ERROR (DbWorker_0) [homeassistant.components.logbook.processor] Error with zha describe event for zha_event
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/logbook/processor.py", line 247, in _humanify
    data = describe_event(event_cache.get(row))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/logbook.py", line 44, in async_describe_zha_event
    zha_device = async_get_zha_device_proxy(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1014, in async_get_zha_device_proxy
    zha_gateway_proxy = get_zha_gateway_proxy(hass)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 993, in get_zha_gateway_proxy
    raise ValueError("No gateway object exists")
ValueError: No gateway object exists

Additional information

No response

[home-assistant[bot]]

Hey there @dmulcahey, @adminiuga, @puddly, @thejulianjes, mind taking a look at this issue as it has been labeled with an integration (zha) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `zha` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign zha` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

zha documentation zha source _{^{(message by IssueLinks)}}

[tbrausch]

I'm having the exact same problem. Once I upgade to 2024.8.x all of my Zigbee devices are unavailable. If I power off/on my Philips Hue lights I can get them to work again, but no luck with all of the Aqara devices. In addition, I did a ZHA backup prior to the installing the HA upgrade. I then tried to reload the ZHA backup and it reported an error. This same backup was verified to work in HA 2024.7.x

[dmulcahey]

Folks we need debug logs to see what is up. Enable debug logging on ZHA, upgrade HA again and then after things fail disable debug logging and attach the downloaded log here. If the integration isn’t running after the upgrade and you can’t disable debug mode just attach the entire HA log here.

[tbrausch]

I've attached the log file here ZHA backup 2024-08-18T17-56-20.676Z.json Thanks for your help and let me know if you need anything else.

[dmulcahey]

Nope, that is a diagnostics file. I need you to do what I outlined in the previous post...

[tbrausch]

Sorry, I clicked on the wrong file to upload. Is this better? home-assistant_zha_2024-08-18T20-09-12.307Z.log

[Mark612]

Here is what happens on a reload:

2024-08-18 17:22:00.226 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry HubZ Smart Home Controller, s/n: _xxxxx_DD - Silicon Labs for zha Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 604, in async_setup result = await component.async_setup_entry(hass, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/zha/init.py", line 124, in async_setup_entry zha_lib_data: ZHAData = create_zha_config(hass, ha_zha_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1224, in create_zha_config zha_options = CONF_ZHA_OPTIONS_SCHEMA(options.get(ZHA_OPTIONS, {})) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 205, in call return self._compiled([], data) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 549, in validate_dict return base_validate(path, data.items(), out) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 382, in validate_mapping raise er.MultipleInvalid(errors) voluptuous.error.MultipleInvalid: extra keys not allowed @ data['disable_new_entities'] 2024-08-18 17:22:02.262 ERROR (MainThread) [aiohttp.server] Error handling request Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 356, in data_received messages, upgraded, tail = self._request_parser.feed_data(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "aiohttp/_http_parser.pyx", line 563, in aiohttp._http_parser.HttpParser.feed_data aiohttp.http_exceptions.BadStatusLine: 400, message: Invalid method encountered:

b'\x16\x03\x01'
  ^

[puddly]

@Mark612 Can you post your Home Assistant config, specifically the zha section?

[Mark612]

@Mark612 Can you post your Home Assistant config, specifically the zha section?

config.txt

[puddly]

I wonder where this config is coming from?

    "config_entry": {
      "data": {
        "device": {
          "baudrate": 57600,
          "path": "/dev/ttyUSB1",
          "flow_control": null
        },
        "radio_type": "ezsp"
      },
      "disabled_by": null,
      "domain": "zha",
      "entry_id": "087ff3f5076411eba92f81ca128d68a2",
      "minor_version": 1,
      "options": {
        "custom_configuration": {
          "zha_options": {
+           "disable_new_entities": false,
            "enable_identify_on_join": true,
            "consider_unavailable_battery": 21600,
            "consider_unavailable_mains": 7400,
            "default_light_transition": 0,
            "enhanced_light_transition": false,
            "light_transitioning_flag": true,
            "always_prefer_xy_color_mode": true,
            "group_members_assume_state": true
          }
        }
      },
      "pref_disable_new_entities": false,
      "pref_disable_polling": false,
      "source": "user",
      "title": "HubZ Smart Home Controller, s/n: 314004DD - Silicon Labs",
      "unique_id": null,
      "version": 4

[Mark612]

It could be from the Z-Wave. The controller does both Z-Wave and Zigbee protocols.

[GregoryPye]

Same issue, though more intermittent, in case another source of data useful.

[sKalach]

I have exactly the same problem and rolled back. Tried two usb sticks - Conbee II & Zi-Stick. Hardware is Raspberry Pi 5.

[puddly]

Please attach debug logs and diagnostic information. Without this, there's not much that can be done to fix this problem.

[sKalach]

Sure, my apologies. I expected that you already had the log from the ticket owner.

• Enabled the Log
• Ran update from 2024.7.3 to 8.2 again
• Stopped the Log
• ZHA had the error for couple of minutes
• And then it is working

The only difference I did is to commented out two last lines in ZHA config section – that device is removed from the network (Climate). Nothing else was changed unless the same 2024.8.2 version has changed in the last four days. I installed the new update on August 16th and could not make it working for three days.

Or during initialization it could not connect to the custom missing climate device.

Please see the file attached.

zha:

custom_quirks_path: /config/custom_zha_quirks/

zigpy_config:

network:

  channel: 24             # What channel the radio should try to use.

  channels: [24, 25]  # Channel mask

device_config:

  - 08:6b:d7:ff:fe:6e:49:00-1:    # format: {ieee}-{endpoint_id}

    type: "switch"              # corrected device type

  - 08:6b:d7:ff:fe:6e:5f:fa-1:

    type: "switch"              # corrected device type

- 00:22:a3:00:00:09:18:28-1:

type: "Climate" # corrected device type

From: puddly @.> Sent: Tuesday, August 20, 2024 2:05 PM To: home-assistant/core @.> Cc: Slawa Kalach @.>; Comment @.> Subject: Re: [home-assistant/core] 2024.8.2 ZHA Fails (Issue #124160)

Please attach debug logs and diagnostic information. Without this, there's not much that can be done to fix this problem.

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/124160#issuecomment-2299442945 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ANK43BVYCNL3Z77RHD2YDQTZSOAORAVCNFSM6AAAAABMWNNTGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJZGQ2DEOJUGU . You are receiving this because you commented. https://github.com/notifications/beacon/ANK43BUFG67LKJSPUVA5HQLZSOAORA5CNFSM6AAAAABMWNNTGCWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUJB23QC.gif Message ID: @. @.> >

[puddly]

@sKalach I think GitHub doesn't accept files over email, unfortunately. Could you attach it to a comment? Thanks!

[sKalach]

home-assistant_zha_2024-08-20T18-27-55.951Z.log config_entry-zha-ab1d89356502e2387a1bdb2a81370ba5.json

[Sfre12]

Yesterday I startet Upgrade from 2024.7.4 to 2024.8.2. And now I have the same problem... Will restore to old backup

[puddly]

@Sfre12 Please attach debug logs.

[martin3000]

Can you see your zigbee USB stick? Maybe by issuing the command "lsusb" on the command line? What installation method did you use for HA? HAOS? Or native core?

[waltailji]

i've been having the same issues with my zigbee network and had to roll back my home assistant version twice (i tried 2024.8.0 and 2024.8.2). the issue only occurs with my battery operated devices, mainly aqara branded buttons, temperature sensors, and leak detectors. all of my mains powered zigbee devices, like smart plugs and light bulbs, still show up fine. if i try to readd my aqara devices they get stuck in "configuring" mode. when i roll back, even after a few hours on 2024.8, my whole zigbee network eventually heals and shows back up. usb stick is a nortek-branded sigma designs HUSBZB-1. my home assistant installation is running on the "home assistant blue" o-droid n2+.

[stevied65]

Same problem

[dam57950]

Same problem, ha web interface sort of reload every few minutes with "disconnected" at the bottom left of the screen and ZHA constantly try to initialize every time it come back to life but can't do it most of the time and even if it can, it's only for little time before it goes down again

[puddly]

@stevied65 @dam57950 Please attach debug logs.

[dam57950]

---

---

---

---

---

---

---

---

---

---

[Sfre12]

@Sfre12 Please attach debug logs.

Hey. I already switched back to the latest Backup from before and it is running again. That is the reason I have no Debug file

[Sfre12]

Can you see your zigbee USB stick? Maybe by issuing the command "lsusb" on the command line? What installation method did you use for HA? HAOS? Or native core?

I am Using a sonoff zigbee bridge, connected by wifi to my HA I have HAOS After restoring to the last backup all is running very well

[Mark612]

I tried 9.1, get the same errors. So what changed from version 7? Why am I one of the few that have this problem?

File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1874, in async_setup raise OperationNotAllowed( homeassistant.config_entries.OperationNotAllowed: The config entry 'HubZ Smart Home Controller, s/n: 3xxx04DD - Silicon Labs' (zha) with entry_id '087xxxxxxd68a2' cannot be set up because it is in state ConfigEntryState.SETUP_ERROR, but needs to be in the ConfigEntryState.NOT_LOADED state

[martin3000]

I see a lot of different errors here with different reasons. Just because your zigbee does not run after an update does not mean that it is always the same error!

[puddly]

@Mark612 Please do not edit the issue title, I changed it because "fails to start up" is extremely vague. The integration can fail to start up for a multitude of reasons and yours fails due to a specific problem.

[Mark612]

What options do I/we have to not get further behind on releases? I'm still running on 2024.7. Is there an ability to run the 2024.7 ZHA architecture on 2024.9?

[Mark612]

Huge appreciation puddly and team. Thank you!