503 or 429 error from Hue bridge - its probably overloaded. Giving up.

jcontrady commented 2 months ago

The problem

When turning on a hue light from an entities card on a dashboard, I received the error message below and the light does not turn on. Light states in HA update to display correctly when they are changed from the native Hue App. Automations issuing light.turn_on or light.turn_off actions also fail.

Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.

A few attempts to restore the Hue integration function

Reload the integration

Failed to reload
Cleaned up the Hue bridge. Then reload the integration

Failed to reload
Removed and added the Hue integration.

Hue bridge auto-detected and added all expected devices, services, and entities. Same error message when attempting to turn on the light again.

Rolled back to 2024.9.1 and the Hue integration works again.

What version of Home Assistant Core has the issue?

core-2024.9.2

What was the last working version of Home Assistant Core?

core-2024.9.1

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Philips Hue

Link to integration documentation on our website

https://www.home-assistant.io/integrations/hue/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-09-24 16:16:43.267 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139661446639920] Unexpected exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 128, in async_request_call
    return await task(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/lights.py", line 102, in set_state
    await self.update(id, update_obj)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/base.py", line 158, in update
    await self._bridge.request("put", endpoint, json=data)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/__init__.py", line 184, in request
    raise BridgeBusy(
aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service
    response = await hass.services.async_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 996, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 626, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/hue/v2/light.py", line 254, in async_turn_on
    await self.bridge.async_request_call(
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 137, in async_request_call
    raise HomeAssistantError(msg) from err
homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.
2024-09-24 16:19:50.375 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:19:50.377 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:19:50.615 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139661446639920] Unexpected exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 128, in async_request_call
    return await task(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/lights.py", line 102, in set_state
    await self.update(id, update_obj)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/base.py", line 158, in update
    await self._bridge.request("put", endpoint, json=data)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/__init__.py", line 184, in request
    raise BridgeBusy(
aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service
    response = await hass.services.async_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 996, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 626, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/hue/v2/light.py", line 254, in async_turn_on
    await self.bridge.async_request_call(
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 137, in async_request_call
    raise HomeAssistantError(msg) from err
homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.
2024-09-24 16:19:51.138 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 3.25 seconds
2024-09-24 16:19:52.381 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:19:52.387 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:19:53.077 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 0.5 seconds
2024-09-24 16:19:53.582 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 0.75 seconds
2024-09-24 16:19:54.337 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 1.0 seconds
2024-09-24 16:19:54.576 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:19:54.576 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:19:54.579 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 3.5 seconds
2024-09-24 16:19:55.023 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 5.25 seconds
2024-09-24 16:19:55.343 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 1.25 seconds
2024-09-24 16:19:56.582 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:19:56.582 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:19:56.598 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 1.5 seconds
2024-09-24 16:19:58.145 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 1.75 seconds
2024-09-24 16:19:58.186 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 3.75 seconds
2024-09-24 16:19:58.588 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:19:58.588 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:19:59.900 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 2.0 seconds
2024-09-24 16:20:00.279 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 5.5 seconds
2024-09-24 16:20:00.684 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:20:00.685 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:20:01.905 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 2.25 seconds
2024-09-24 16:20:02.034 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 4.0 seconds
2024-09-24 16:20:02.692 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:20:02.692 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:20:02.976 INFO (MainThread) [aiohue.v2[X.X.X.X]] Connection to bridge closed.
2024-09-24 16:20:04.377 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 0.5 seconds
2024-09-24 16:20:04.789 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Connected to EventStream
2024-09-24 16:20:04.789 DEBUG (MainThread) [aiohue.v2[X.X.X.X].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-09-24 16:20:04.883 DEBUG (MainThread) [aiohue.v2[X.X.X.X]] Got 503 or 429 error from Hue bridge, retry request in 0.75 seconds

Additional information

No response

home-assistant[bot] commented 2 months ago

Hey there @balloob, @marcelveldt, mind taking a look at this issue as it has been labeled with an integration (hue) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `hue` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign hue` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

hue documentation hue source _{^{(message by IssueLinks)}}

theodric commented 1 month ago

I'm not sure if you welcome "me too!" messages, but I'm having a fault of the exact same description which led me to this issue.

I will note that WRITING commands from HA to the Hue hub fails with the 'Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.' message provided by jcontrady

However, READING from the Hue hub into HA still seems to work-- my motion sensors still happily report events, temperature, and light levels, and if I switch a light on with some other tool (the official Android app, or Hue Essentials, or Hue Menu Bar on the Mac, or Hoppla on Linux), the device's status change is reflected in HA.

The Hue Hub is a BSB002 running firmware 1.66.1966117030, which hueblog.de indicates was released sometime around 23 September, corresponding roughly with the timing of this issue's presentation. I bet Philips b0rked something.

Restoring from my July backup did not help. Deleting the Hue integration and re-pairing with the hub did not help. If you want any logs or troubleshooting from me, I'm happy to help, otherwise I'll go away.

jcontrady commented 1 month ago

Hi @theodric, Thanks for mentioning the Hue hub model and firmware. I have the same BSB002 on firmware 1.66.1966117030 and noticed the same behavior that reading devices into HA works. Not sure I grabbed the correct logs so if you want to add yours I can't imagine it would hurt.

ardichoke commented 1 month ago

I'm also having this same issue, started for me in core 2024.10.01. My Hue bridge is also on the same firmware version.

albaintor commented 1 month ago

Same problem here with firmware 1.67.1967054010 What is strange is that I have a second hub with the same firmware but this one is still working

petos-hue commented 1 month ago

Same problem, started for me in core 2024.10.01. Hue Bridge model BSB002, SW Version: 1.66.1966117030 from 24.09.2024.

Removing the Hue integration from HA and re-adding it helps, but after a few hours it stops working again.

LOG from Hue Bridge Integration in HA: `homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up. 2024-10-09 08:44:48.770 DEBUG (MainThread) [aiohue.v2[192.168.50.126]] Got 503 or 429 error from Hue bridge, retry request in 2.0 seconds 2024-10-09 08:44:49.082 DEBUG (MainThread) [aiohue.v2[192.168.50.126]] Got 503 or 429 error from Hue bridge, retry request in 6.0 seconds 2024-10-09 08:44:50.088 DEBUG (MainThread) [aiohue.v2[192.168.50.126].events] Connected to EventStream 2024-10-09 08:44:50.092 DEBUG (MainThread) [aiohue.v2[192.168.50.126].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds 2024-10-09 08:44:50.326 DEBUG (MainThread) [aiohue.v2[192.168.50.126]] Got 503 or 429 error from Hue bridge, retry request in 4.5 seconds 2024-10-09 08:44:50.654 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139940847481136] Unexpected exception Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 128, in async_request_call return await task(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/lights.py", line 102, in set_state await self.update(id, update_obj) File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/base.py", line 158, in update await self._bridge.request("put", endpoint, json=data) File "/usr/local/lib/python3.12/site-packages/aiohue/v2/init.py", line 184, in request raise BridgeBusy( aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service response = await hass.services.async_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call response_data = await coro ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service return await target(service_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/adaptive_lighting/hass_utils.py", line 62, in service_func_proxy await existing_service.job.target(call) File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 996, in entity_service_call single_response = await _handle_entity_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call result = await task ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/light/init.py", line 628, in async_handle_light_on_service await light.async_turn_on(*filter_turn_on_params(light, params)) File "/usr/src/homeassistant/homeassistant/components/hue/v2/light.py", line 255, in async_turn_on await self.bridge.async_request_call( File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 137, in async_request_call raise HomeAssistantError(msg) from err homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up. 2024-10-09 08:44:50.779 DEBUG (MainThread) [aiohue.v2[192.168.50.126]] Got 503 or 429 error from Hue bridge, retry request in 2.25 seconds 2024-10-09 08:44:51.143 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139940847481136] Unexpected exception Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 128, in async_request_call return await task(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/lights.py", line 102, in set_state await self.update(id, update_obj) File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/base.py", line 158, in update await self._bridge.request("put", endpoint, json=data) File "/usr/local/lib/python3.12/site-packages/aiohue/v2/init.py", line 184, in request raise BridgeBusy( aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.`

theodric commented 1 month ago

Bizarrely, after I experienced a two-hour power outage, thereby fully power-cycling everything all at once, the Hue integration magically started working again. ¯_(ツ)_/¯

ardichoke commented 1 month ago

Bizarrely, after I experienced a two-hour power outage, thereby fully power-cycling everything all at once, the Hue integration magically started working again. ¯(ツ)/¯

Did it stay working? I could get mine working again just by restarting Home Assistant core, but it would stop working again after a little while with the same errors. Haven't tried 10.2 or 10.3 yet though.

ardichoke commented 1 month ago

Tried 2024.10.3, issue still happens there. Downgraded again to 2024.9.3 and it went back to working without issue.

cmnitz commented 1 month ago

Same issue here with 2024.10.3. Downgrade to 2024.9.3 and works without issue. Using this command: ha core update --version=2024.9.3 https://community.home-assistant.io/t/how-to-downgrade-home-assistant/179735

TC1666 commented 1 month ago

This apparently is a widespread issue, added a new generic Tuya color bulb to my Hue bridge this morning, commands were sporadic and the bridge became non-responsive at times. Downgrade HA Core to 2024.9.3 and works without issue, didn't even have to restart the Hue bridge.

theodric commented 1 month ago

Bizarrely, after I experienced a two-hour power outage, thereby fully power-cycling everything all at once, the Hue integration magically started working again. ¯(ツ)/¯

Did it stay working? I could get mine working again just by restarting Home Assistant core, but it would stop working again after a little while with the same errors. Haven't tried 10.2 or 10.3 yet though.

I think it's been about ten days now and it's still working (again) for me post-power outage. After it magically started working, I upgraded my HAOS aarch64 VM to core 2024.10.4, Supervisor 2024.10.3, and OS 13.2, and the Hue integration still functions.

ardichoke commented 1 month ago

Bizarrely, after I experienced a two-hour power outage, thereby fully power-cycling everything all at once, the Hue integration magically started working again. ¯(ツ)/¯

Did it stay working? I could get mine working again just by restarting Home Assistant core, but it would stop working again after a little while with the same errors. Haven't tried 10.2 or 10.3 yet though.

I think it's been about ten days now and it's still working (again) for me post-power outage. After it magically started working, I upgraded my HAOS aarch64 VM to core 2024.10.4, Supervisor 2024.10.3, and OS 13.2, and the Hue integration still functions.

You seem to have gotten really lucky. I powered my haos VM off and unplugged my hue bridge, then powered both back up. After a few hours, it went right back to a non functional state again on 2024.10.4.

marcelveldt commented 1 month ago

I cant reproduce this issue at all. The only way to trigger this is if there are MANY commands to the Hue hub. At some point its (very limited) webserver will start denying requests.

Do you have other apps connected to the Hue bridge like Hue Essentials or node Red ? If so, try shutting them down for a while
Do you have any integrations or automations that do many commands to Hue lights such as Adaptive lightning ? If so, try disabling that for a while to test.

This may as well have some other external cause such as an HTTP connection not being closed properly etc. That will need to be closely looked at.

ardichoke commented 1 month ago

Obviously, I cannot speak for anyone else here, but I can answer these for my experience.

* Do you have other apps connected to the Hue bridge like Hue Essentials or node Red ?
  If so, try shutting them down for a while

I only have two things talking to Hue, Home Assistant and Google Assistant. I rarely use Google Assistant to interact with my Hue lights, but even when Home Assistant has been throwing these "giving up" errors, I have been able to control my Hue lights via Google Assistant (and via the Hue app). This certainly makes it seem like the problem is on the HA side.

* Do you have any integrations or automations that do many commands to Hue lights such as Adaptive lightning ?
  If so, try disabling that for a while to test.

No. The only interactions that HA does with Hue is turning on Hue scenes and turning Hue lights off. Nothing more.

This may as well have some other external cause such as an HTTP connection not being closed properly etc. That will need to be closely looked at.

For me, at least, this is solely happening when I update HA core to 2024.10.1 or later. If I'm running 2024.9.3 or earlier, I have no issues. I don't know if something changed in the core that would impact requests sent to the Hue bridge, or if this is pure coincidence. I can say I've upgraded HA core to each of the 2024.10 versions when they came out to test and every time this started happening, after which I would downgrade to 2024.9.3 and the integration would work as expected again, so it definitely seems to be consistent and repeatable in my install at least.

marcelveldt commented 1 month ago

Thanks for the additional info - I'll have to investigate if there have been any changes to our http client then (aiohttp).

ardichoke commented 1 month ago

If there's anything I can provide to help troubleshoot this, please let me know since I can reliably cause the issue to occur.

k-masters commented 4 weeks ago

I just wanted to chime in and say that I also ran into this issue. Hub is running 1.67.1967054020 and HA is on 2024.10.2.

Best I can tell is that it started 3 days ago based off the last time one of my automations turned on a light.

As with @theodric, reading the state from the hub worked, but writing did not. A reboot of the hub seems to have fixed it for now at least.

Here is the log from HA, but it doesn't look to be anything particularly useful


Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service
response = await hass.services.async_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
response_data = await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
return await target(service_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/homeassistant/__init__.py", line 151, in async_handle_turn_service
await asyncio.gather(*tasks)
File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
response_data = await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
return await target(service_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1021, in entity_service_call
raise result from None
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1610, in async_request_call
return await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call
result = await task
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 628, in async_handle_light_on_service
await light.async_turn_on(**filter_turn_on_params(light, params))
File "/usr/src/homeassistant/homeassistant/components/hue/v2/light.py", line 255, in async_turn_on
await self.bridge.async_request_call(
File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 137, in async_request_call
raise HomeAssistantError(msg) from err
homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.
2024-10-29 13:55:17.553 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 477, in _handle_request
resp = await request_handler(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 559, in _handle
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 117, in impl
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 210, in forwarded_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 85, in ban_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 242, in auth_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 32, in headers_middleware
response = await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/http.py", line 73, in handle
result = await handler(request, **request.match_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/diagnostics/__init__.py", line 280, in get
data = await info.config_entry_diagnostics(hass, config_entry)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/hue/diagnostics.py", line 23, in async_get_config_entry_diagnostics
return await bridge.api.get_diagnostics()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohue/v2/__init__.py", line 276, in get_diagnostics
full_state = await self.request("get", "clip/v2/resource")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohue/v2/__init__.py", line 184, in request
raise BridgeBusy(
aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.```

ardichoke commented 2 weeks ago

Any updates on this? I upgraded to 2024.11.01 and had this happen again afterwards. I'm wondering if I should try removing and adding my hue bridge again, but its really annoying having to set up my automations again...

Based on my own experiences only, it looks like aiohttp got updated from 3.10.5 to 3.10.8 between the working and problematic releases. This is the diff between those versions: https://github.com/aio-libs/aiohttp/compare/v3.10.5...v3.10.8

cmnitz commented 1 week ago

Any updates on this? I upgraded to 2024.11.01 and had this happen again afterwards. I'm wondering if I should try removing and adding my hue bridge again, but its really annoying having to set up my automations again...

FWIW I upgraded from 2024.9.3 to 2024.11.1 and haven’t had the issue since.

ardichoke commented 1 week ago

Still persists for me in 2024.11.2. I grabbed a debug log today of it happening, but I don't think there's anything particularly useful in it. Just a lot of this:

2024-11-19 17:41:51.144 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 5.5 seconds
2024-11-19 17:41:51.250 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:51.250 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-11-19 17:41:51.496 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:51.496 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-11-19 17:41:51.500 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:51.500 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-11-19 17:41:51.729 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 5.5 seconds
2024-11-19 17:41:51.767 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:51.768 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-11-19 17:41:52.649 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 3.5 seconds
2024-11-19 17:41:53.364 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:53.365 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Disconnected from EventStream - Reconnect will be attempted in 2 seconds
2024-11-19 17:41:53.678 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:53.701 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:53.780 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:55.480 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Connected to EventStream
2024-11-19 17:41:56.260 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 3.75 seconds
2024-11-19 17:41:56.754 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 5.75 seconds
2024-11-19 17:41:57.335 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 5.75 seconds
2024-11-19 17:42:00.017 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 4.0 seconds
2024-11-19 17:42:02.510 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 6.0 seconds
2024-11-19 17:42:03.091 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 6.0 seconds
2024-11-19 17:42:04.023 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 4.25 seconds
2024-11-19 17:42:08.279 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 4.5 seconds
2024-11-19 17:42:08.515 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 6.25 seconds
2024-11-19 17:42:09.097 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 6.25 seconds
2024-11-19 17:42:12.785 DEBUG (MainThread) [aiohue.v2[172.16.10.50]] Got 503 or 429 error from Hue bridge, retry request in 4.75 seconds
2024-11-19 17:42:14.773 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Error while sending keepalive: 25 requests to the bridge failed, its probably overloaded. Giving up.
2024-11-19 17:42:15.350 DEBUG (MainThread) [aiohue.v2[172.16.10.50].events] Error while sending keepalive: 25 requests to the bridge failed, its probably overloaded. Giving up.

If desired, I can provide the full log, but I don't see anything all that useful.

I do see occasional 503 or 429 messages throughout the day, but normally after a retry or two it is successful. Eventually, something tips and it starts hitting the 25 retry, giving up state at which point it does not ever recover.

I also tried rebooting the Hue Bridge, to see if that would restore functionality, but even after that Home Assistant is still doing the same thing, which further suggests that the problem is on the Home Assistant side of the equation.

It makes me wonder if the retries are too aggressive, or if something is continuing to send requests to the bridge in the background which eventually leads to a cascading failure.

funkyHat commented 1 week ago

I don't have good evidence to support this theory, because I didn't notice things were working straight away, but I also had this issue for a while.

I separately noticed that my HA wall time was wrong by a few minutes, which I resolved by installing ntpd on my docker host.

Some time later I noticed Hue devices were all showing up normally again. Could a time difference between the Hue bridge & HA cause something like this?

I don't really want to switch back to systemd-timesyncd which doesn't keep my system clock in time just to confirm this, sorry 😅

marcelveldt commented 1 day ago

Today I was (finally!) able to reproduce this issue after adding a lot more lights to one single Hub. So this seems to be a combination of factors but it is has to do with the amount of resources (lights/devices/scenes etc) on the Hue hub - makes sense as that will also increase the json payloads. This may as well be as simple as a timeout or message size limit. In my case, a restart of the Hue bridge resolved the issue. I'll poke more in to this tomorrow.

Question to you all is how many resources you have on the hue bridge ? So lights, zones, rooms, scenes and remotes etc. Give or take, doesnt have to be precise. Just to have an indication if this only happens on large setups.

albaintor commented 1 day ago

I do have a large setup : about 50 bulbs and 15 devices

ardichoke commented 1 day ago

I have a pretty small Hue setup... 8 bulbs, 4 rooms, 1 zone, 28 scenes (this is just the "natural light" scene in each of the 4 rooms)

itSme1990 commented 1 day ago

I do have a mid-size setup with 24 lamps, 3 motion sensors and 2 dial switches. All connected to on one hub.
All in all the hue Integration in Home assistant has 30 devices, 10 services and 155 entities.

marcelveldt commented 15 hours ago

Based on my own experiences only, it looks like aiohttp got updated from 3.10.5 to 3.10.8 between the working and problematic releases. This is the diff between those versions: https://github.com/aio-libs/aiohttp/compare/v3.10.5...v3.10.8

Yeah, I noticed that as well and its actually the only change that could possibly explain the behavior. I'm currently trying to reproduce the issue with and without that aiohttp bump involved to pinpoint it. So far I was not able to reproduce with downgrading aiohttp.

k-masters commented 10 hours ago

After the hub reboot way back when I made my last comment sometime last month I haven't had the issue pop back up. I'm still running HA 2025.10.2.

As far as my Hue setup, I'd say it's rather small. 10 bulbs, 1 switch, 4 rooms, and 20 or so scenes of which most are the default ones from the Hue app.

My Hue integration has 11 devices, 4 services, and 66 entities.

c4bler commented 9 hours ago

I'm also experiencing this issue. Press on/off on an individual light in HA, nothing happens. Sometimes 20 seconds later the light will 'catch up' and change state.

5 devices (4 lights + bridge), 3 services, 11 entities

Via HA: Delays, issues as seen from other commenters Via HTTP PUT via Hue API / python requests.put : all works ok. I can spam it with JSON states as many times as I want and it keeps working, even during the failing HA errors.

Hue Bridge: 1.67.1967054020 HA Core: 2024.11.3

Logger: homeassistant.core
Source: core.py:2822
First occurred: 14:26:35 (5 occurrences)
Last logged: 14:27:28

Error executing service: <ServiceCall light.turn_off (c:<somerandomID>): entity_id=['light.mylight1'], params=>
Error executing service: <ServiceCall light.turn_off (c:<somerandomID>): entity_id=['light.mylight2'], params=>
Error executing service: <ServiceCall light.turn_on (c:<somerandomID>): entity_id=['light.mylight2'], params=>
Error executing service: <ServiceCall light.turn_on (c:<somerandomID>): entity_id=['light.mylight1'], params=>
Error executing service: <ServiceCall light.turn_off (c:<somerandomID>): entity_id=['light.mylight1'], params=>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 128, in async_request_call
    return await task(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/lights.py", line 102, in set_state
    await self.update(id, update_obj)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/controllers/base.py", line 158, in update
    await self._bridge.request("put", endpoint, json=data)
  File "/usr/local/lib/python3.12/site-packages/aiohue/v2/__init__.py", line 184, in request
    raise BridgeBusy(
aiohue.errors.BridgeBusy: 25 requests to the bridge failed, its probably overloaded. Giving up.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 2822, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2845, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1007, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1079, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 639, in async_handle_light_off_service
    await light.async_turn_off(**filter_turn_off_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/hue/v2/light.py", line 280, in async_turn_off
    await self.bridge.async_request_call(
  File "/usr/src/homeassistant/homeassistant/components/hue/bridge.py", line 137, in async_request_call
    raise HomeAssistantError(msg) from err
homeassistant.exceptions.HomeAssistantError: Request failed: 25 requests to the bridge failed, its probably overloaded. Giving up.

speed47 commented 5 hours ago

Hitting the same issue (I installed my hue bridge a couple days ago), my setup is rather small: 4 lamps in 1 room, and one hue remote (dimmer switch). Changes from the hue app are immediate and mirrored in HA at once, but actions from HA timeout almost always. I can test integration patches and/or provide logs as needed.

home-assistant / core