Hue integration sends commands to the bridge too fast

eikowagenknecht commented 2 years ago

The problem

Yes, this has already been described here: https://github.com/home-assistant/core/issues/9267

But since that issue is closed and the problem still occurs four years later in HA 2021.11.5 I'm creating this new issue.

The problem:

Turning many (>10 or >20) lights on/off in a short time span, some don't get the command.
This seems to be a problem of the Hue Bridge not being able to consistently handle more that 10 "light" commands or 1 "group" command per second (see https://developers.meethue.com/develop/hue-api-v2/core-concepts/#limitations, needs hue developer account to open)
The result looks like this. In the example I turned all lights off using a scene, but one light "stayed" on. In HA it looks like it correctly switched to off and then on again:

The workaround and it's caveats:

@sdague suggested in the original issue to just use Hue Scenes as a workaround. This way only one command has to be sent to the Hue Bridge.
This has multiple problems:
- It's inconvenient to create a Hue scene in the Hue app just to use it with Home Assistant.
- It does not integrate neatly in the Home Assistant UI, especially when mixing with non-Hue lights.
- Hue scenes define the complete state (color, brightness etc.) of all included lights. As such it's not possible to create a Scene that switches 20 lights ON to their previous state.
- This approach does not work when the scene is dynamic. E.g determine which lights to turn on using some condition
- For rooms with many lights (yes, my kitchen indeed has ~15 lights!) the simple "entities" cards in Lovelace have a "turn on/off all" switch. This can not reliably be used with Hue.

Possible solutions:

Rate limit the commands sent to the hue bridge to 10/s, queuing them in the order they are issued from HA.
Notice that the command fails (maybe the hue bridge gives some kind of feedback / error code when reaching the rate limit?) and resend commands that have failed after a short wait.

What version of Home Assistant Core has the issue?

core-2021.11.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Philips Hue

Link to integration documentation on our website

https://www.home-assistant.io/integrations/hue

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

probot-home-assistant[bot] commented 2 years ago

Hey there @balloob, @marcelveldt, mind taking a look at this issue as it has been labeled with an integration (hue) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

hue documentation hue source _{^{(message by IssueLinks)}}

marcelveldt commented 2 years ago

@eikowagenknecht do you own a Hue bridge V1 or V2 ?

eikowagenknecht commented 2 years ago

@marcelveldt Hue Bridge V2. Some more info about the bridge if needed:

Name | Wert -- | -- apiversion | 1.48.0 backup | {"status":"idle","errorcode":0} bridgeid | ********** datastoreversion | 113 dhcp | true factorynew | false gateway | 192.168.*****.1 internetservices | {"internet":"connected","remoteaccess":"connected","time":"connected","swupdate":"connected"} ipaddress | 192.168.*****.34 linkbutton | false localtime | 2021-12-01T14:28:55 mac | **:**:**:**:**:** modelid | BSB002 name | Philips hue netmask | 255.255.255.0 portalconnection | connected portalservices | true portalstate | {"signedon":true,"incoming":false,"outgoing":true,"communication":"disconnected"} proxyaddress | none proxyport | 0 replacesbridgeid | null starterkitid | swupdate | {"updatestate":0,"checkforupdate":false,"devicetypes":{"bridge":false,"lights":[],"sensors":[]},"url":"","text":"","notify":true} swupdate2 | {"checkforupdate":false,"lastchange":"2021-11-08T13:12:51","bridge":{"state":"noupdates","lastinstall":"2021-11-08T13:01:57"},"state":"noupdates","autoinstall":{"updatetime":"T14:00:00","on":true}} swversion | 1948086000 timezone | Europe/Berlin UTC | 2021-12-01T13:28:55 whitelist | *** zigbeechannel | 15

(Read using another addon (homematic hue addon), if you need values directly from Home Assistant I#ll have to look into it how to get those)

marcelveldt commented 2 years ago

OK, thanks. We've rewritten the Hue integration for Home Assistant entirely to support the new API. This change will most probably (I'm like 99% sure) fix your issues.

The new Hue integration will be available in next HA release (or try the dev version)

marcelveldt commented 2 years ago

BTW: Be aware that the limit of about 10 requests per second counts for all apps connected to the api. So, you're speaking about homematic hue addon. Remember that will negatively impact the performance too.

eikowagenknecht commented 2 years ago

@marcelveldt Nice, that sounds promising. I'll make sure to test the new integration as soon as I can. Still new to Home Assistant (coming from openHAB), so if you could give me a quick push how to use the dev version of the integration that would be great :-)

Good point that the 10/s are shared between apps (in my case openHAB, Homematic addon, various Hue Apps, ...). So the approach of recognizing when a command actually failed and then resending it would probably be more robust.

marcelveldt commented 2 years ago

There is already logic that failed commands get resent but it fails at some point. In the new integration this is a bit more robust and extended but there is still a point where no more retries are attempted.

If your bridge is being queried all the time by all those apps it will very soon be overloaded. Good thing about the new API in Hue is that it no longer needs polling so hopefully for you the OpenHab and Homematic implementations will soon switch to the new API too.

The old implementation in HomeAssistant (and my guess is that your other apps do the same) was/is polling the bridge every 0.5 seconds for updates. This is no longer needed for the V2 api where the bridge streams updates to HA.

marcelveldt commented 2 years ago

As for testing the dev version, there's a somewhat hidden option for that on Home Assistant OS. You can also wait for the beta which will be released within a few days

eikowagenknecht commented 2 years ago

Good to hear that polling will no longer be neccessary. Unfortunately the Homematic addon is no longer developed because the developer just switched to Home Assistant himself. But it can at least be rate limited, currently it's polling "only" every 2 seconds. For the openHAB addon I do not have much hope for a fast solution. One of my main reasons for coming to Home Assistant is indeed the very slow development on the openHAB side. But it is also configurable and currently at 1 call every second.

Anyways, I'll leave this issue open until I can test it with the new version (which shouldn't be that long) if that's ok with you.

marcelveldt commented 2 years ago

If you're eager to test, you could test with the dev version container in docker. Otherwise just wait a few days until beta is released. Release of stable version is december 11th, together with the event

balloob commented 2 years ago

We're tagging a beta version tomorrow.

sirgilliad commented 2 years ago

Hi, Another option is to maintain the state of devices in openhab, and then have rules checking that the state in the devices an OH is the same. That way your command will definetly be executed over a few seconds.

I am looking forward to the new beta.

eikowagenknecht commented 2 years ago

The beta version works great, but this problem still occurs sometimes for me:

One of the lights goes off in HA, but not in reality and some time later HA refreshes to see that it's still on. Then I turn it off again from HA.

marcelveldt commented 2 years ago

The beta version works great, but this problem still occurs sometimes for me:

One of the lights goes off in HA, but not in reality and some time later HA refreshes to see that it's still on. Then I turn it off again from HA.

This data comes directly from the bridge/api, HA does not manually refresh anything. So if it's reported as OFF, Hue is reporting it as OFF. What does the app show at these times ?

Like I said before, you really should try to eliminate all those other connections to your Hue bridge. The thing is very power limited and can only handle a maximum of about 10 connections per second, less if you have many lights/sensors connected to it.It's pretty easy to overload it.

The new Hue integration is a step in the right direction as it will only have one connection open to the bridge (at all times) to receive updates and for each command you send from HA to Hue a new connection is made.

Do you see any errors related to Hue in your HA log ?

BTW: Is the light from the screenshot above a single Hue light or a group/zone ?

marcelveldt commented 2 years ago

Forgot to tell you: Using a Hue zone for multiple lights is way more efficient than sending multiple light commands from HA. When you turn on a Hue zone, this single command is executed on the zigbee stack. If you create a HA group with Hue lights or send multiple lights an ON/OFF command at once, these will all be directed one-by-one

eikowagenknecht commented 2 years ago

Thank you for your detailled replies.

Do you see any errors related to Hue in your HA log ?

No, I don't.

BTW: Is the light from the screenshot above a single Hue light or a group/zone ?

It's from a single light.

Using a Hue zone for multiple lights is way more efficient than sending multiple light commands from HA.

I've switched all "groupable" light to zones now and this indeed makes the behaviour quite more stable.

Like I said before, you really should try to eliminate all those other connections to your Hue bridge.

I finally migrated everything Hue related from most of my other sources to HA. Only Google, Homematic and Home Assistant remain now.

After these local improvements, I could not reproduce the issue, so I'm closing this now as it seems to be a combination of too many lights on my bridge and too many connections to it. Nothing HA can fix. Hopefully there will be a Hue Bridge v3 soon with better hardware.

home-assistant / core