Support for Hue “push” updates

[paolotremadio]

Hi @ebaauw, Philips has finally made available the specs for the API v2. They do support “push notifications” for status updates.

I’ve created an API client that works with the hue bridge: https://github.com/homegraph/philips-hue-push-client

Would you be happy to work together to implement push updates in homebridge-hue?

[ebaauw]

Cool. I did see the API v2 documentation, but haven’t yet started working on it. I don’t think I’ll want to expand Homebridge Hue to support API v2; it’s probably better to split it into a Homebridge Hue2 versus a Homebridge deCONZ plugin.

Also, I’m still waiting on the roll-out by Signify; my bridge still uses a self-signed certificate. Would love to whitelist their root certificate instead and lose the certificate pinning stuff.

[paolotremadio]

I agree, it might be good to split the project as the API will start diverging. Also deCONZ will eventually have a v2 api too.

I'll close this for now, we can always reopen this when the time comes. My bridge is also still working on self signed certificate.

[ebaauw]

@paolotremadio were you actually able to get something useful from the event stream?

I managed to parse light state updates and convert these back to API v1 format, so they could be used in Homebridge Hue. However, I hardly see any benefit over polling the bridge. The update rate is still limited by the bridge polling the lights. With more than one light, it will take seconds or longer before the bridge will have next poleed the light and send the corresponding event (try setting "effect": "colorloop" on the light).

I'm getting events on button presses as well, but no clue in the event which button is pressed. In other words, I can figure out the last digit of the buttonevent value (x000 vs x001 vs x002 vs x003), but not the first digit (100y vs 200y vs 300y vs 400y). I suppose that might be resolved through the API v2 rid value, but then I need to implement full API v2 support to map the rid value to the button.

I'm not getting any events when turning the dial of the Lutron Aurora.

[paolotremadio]

Groups and lights get sent pretty quickly (as soon as Hue polls the lights).

Sensors / buttons are sent almost instantaneously. You get all the type of event and the button that has been pressed; you’re right, you need to know the rid to make sense of which button it is.

The api v2 does expose the IDs for API v1 resources so you could use that to create a mapping.

One of the API v2 endpoints gets you all the resources available (similar to /api/username/ that gets you all the bridge data). I think it’s called “resources”, I can check that for you.

The main advantage of API v2 is buttons/sensors events (because they’re time sensitive)

[ebaauw]

One of the API v2 endpoints gets you all the resources available (similar to /api/username/ that gets you all the bridge data). I think it’s called “resources”, I can check that for you.

There is an API endpoint just to get all buttons, https://developers.meethue.com/develop/hue-api-v2/api-reference/#resource_button. I'll see if I can call that one and build a map on startup.

The main advantage of API v2 is buttons/sensors events (because they’re time sensitive)

Should be. However, on https://developers.meethue.com/develop/hue-api-v2/migration-guide-to-the-new-hue-api/:

Currently there is a 1 second rate limit on the amount of event containers the Bridge will send. If the same property has changed twice within that timeframe, you only get the last state. If multiple resources have changed within that timeframe, you will get multiple events grouped in a single container.

I'm seeing a delay between the initial_press (x000) and repeat (x001) events.

[ebaauw]

Beta v0.13.31-0 contains first step to support the event stream notifications:

• Added EventStreamClient class. Connects to the event stream and converts events into API v1 compatible notifications, similar to deCONZ web socket notification.
• Added ph eventlog command that now works for deCONZ as well as for Hue bridge with API v2;
• Deprecate dc_eventlog command.

Note that Homebridge Hue itself doesn't yet connect to the event stream. I want to run ph eventlog for a while to see if anything "weird" pops up.

[paolotremadio]

You can go to /clip/v2/resource/device and scan for known sensors.

E.g.

        {
            "id": "4ee10f86-d71b-4345-b8cc-02b3fea28f99",
            "id_v1": "/sensors/3",
            "metadata": {
                "archetype": "unknown_archetype",
                "name": "Living Room Dimmer"
            },
            "product_data": {
                "certified": true,
                "manufacturer_name": "Signify Netherlands B.V.",
                "model_id": "RWL021",
                "product_archetype": "unknown_archetype",
                "product_name": "Hue dimmer switch",
                "software_version": "1.1.28573"
            },
            "services": [
                {
                    "rid": "c1b29ac9-20df-4f29-96a6-88154f1ad28a",
                    "rtype": "button"
                },
                {
                    "rid": "ec39d230-f4d2-44b4-9129-7a181da9baf4",
                    "rtype": "button"
                },
                {
                    "rid": "6945092b-dfaf-4cde-9bf6-22b949289a44",
                    "rtype": "button"
                },
                {
                    "rid": "c0194605-c1bb-457f-8f24-e9c93ba755f6",
                    "rtype": "button"
                },
                {
                    "rid": "04fecce3-bcad-4ae5-bbfa-77db5a6cb492",
                    "rtype": "device_power"
                },
                {
                    "rid": "0f6cd45f-1bec-49e5-933d-a09d69f759c4",
                    "rtype": "zigbee_connectivity"
                }
            ],
            "type": "device"
        }

The buttons are presented in order:

• Button 1 - rid c1b29ac9-20df-4f29-96a6-88154f1ad28a
• Button 2 - rid ec39d230-f4d2-44b4-9129-7a181da9baf4
• etc...

The alternative is to go to /clip/v2/resource/button, scan for known sensors and use the metadata.control_id as button id.

[paolotremadio]

You can also use this polyfill for the EventSource client: https://www.npmjs.com/package/eventsource

[ebaauw]

The alternative is to go to /clip/v2/resource/button, scan for known sensors and use the metadata.control_id as button id.

Already doing that, see above commit.

[paolotremadio]

Great. I had to manually configure the buttons rid in my home automation system. It is a bit annoying but I guess they’ve changed it in order to open the door for more complex kind of controllers.

The new API is overall a bit improvement, the values are more readable (eg brightness in % rather than 0…254). The button part could be considered a step back but I guess it’s just a matter of changing implementations

[ebaauw]

The new API is overall a bit improvement

I beg to differ. It's a typical Hue / Signify implementation, which, imho, sucks big time.

• No lastupdated, so I need to use the event time for this. Upon the next poll, sometimes lastupdated is actually a second earlier than the event, causing duplicate button events.
• The API v1 uses ZGP command codes as buttonevent values for ZGP switches, but these values cannot be derived from the events. Had to do some changes to accomodate that as well.
• Brightness in percentage with two decimals? Really?
• Not support for hue/sat or worse: colorloop (you need to set hue/sat to set the start of the colorloop).
• No support for dark nor daylight of the motion sensor's light level sensor, effectively rendering the event useless, as you'd trigger automations on these, instead of on lightlevel.
• The naming is inconsistent: dimming vs brightness, color vs xy.x and xy.y, color_temperature vs mirek, light vs light_level. And status and enabled take a simple value instead of an obejct.

Anyway, I think I got it working in beta v0.13.31-2. Just hope I didn't break anything dealing with these peculiarities.

[ebaauw]

Issues with v0.13.31-2:

• Plugin issues TypeError on unknown events [fixed];
• Plugin doesn't handle button events for Hue tap [fixed];
• EventStreamClient doesn't recognise when connection to the Hue bridge has been lost and doesn't reconnect.

[ebaauw]

Released v0.13.31, which should reconnect automatically and includes all fixes.

[roboterm]

When entering ph -H x.x.x.x. -D eventlog I’m getting ph: error: eventlog: unknown command

(I substituted the IP address of my Hue bridge of course)

[ebaauw]

Are you running an older version? What’s the output of ph -V?

[roboterm]

0.13.30

[ebaauw]

ph eventlog was introduced in v0.13.31, see the release notes.

[roboterm]

Wed Dec 22 2021 09:06:05] request 1: GET /resource/button [Wed Dec 22 2021 09:06:06] request 1: 200 OK [Wed Dec 22 2021 09:06:06] request 2: GET /eventstream/clip/v2 [Wed Dec 22 2021 09:06:06] request 2: 200 OK [Wed Dec 22 2021 09:06:06] listening on https://192.168.2.100/eventstream/clip/v2

[ebaauw]

That’s looking good. You should see events when you control your lights (e.g from the Hue app) or when you trigger a button or sensor. I’m especially interested in events that aren’t yet parsed, but might contain relevant data.

[roboterm]

But v0.13.31 prevents Homebridge starting.

[ebaauw]

That’s not good. Please attach the debug log file.

[roboterm]

Is that enough for you?

[ebaauw]

No, of course not.

The EPROTO indicates an issue with SLL support on your system. Nothing to do with Homebridge Hue. Possibly related to a "weird" installation of NodeJS, or another Homebridge plugin breaking SSL support (wouldn't be the first time).

v0.13.32 won't try and connect to the event stream when forceHttp has been set. Note that the Hue API v2 no longer supports plain http as a fallback.

[roboterm]

Aha ok. Thank you 🙏

It works now.

I upgraded node via sudo hb-config.

[jostrasser]

Hi @ebaauw I've updated to the latest version of homebridge-hue. Is there anything to do that I can use the new push implementation? Any Homebridge configuration (json config) change needed to activate it? Right now I cannot see any speed improvements with my Hue Bridge V2 / Dimmer Switches / Tap Switches. Thanks!

[ebaauw]

Homebridge Hue should use the event stream automatically, provided the Hue bridge has the right firmware version and that you connect to it over HTTPS.

[jostrasser]

Homebridge Hue should use the event stream automatically, provided the Hue bridge has the right firmware version and that you connect to it over HTTPS.

Hi @ebaauw thanks!

The firmware should be the newest available I think (1.48.1948086000) Do I have to set anything for https? Or will it connect automatically via https? How can I check if https is used?

Thanks!

[ebaauw]

Unless you’ve set forceHttp, https should be used by default. Check using netstat: you should see a connection to port 443 on the bridge, instead of to port 80.

Did you check the Homebridge log? There should be a message when Homebridge Hue connects to the event stream.

[jostrasser]

Unless you’ve set forceHttp, https should be used by default. Check using netstat: you should see a connection to port 443 on the bridge, instead of to port 80.

Did you check the Homebridge log? There should be a message when Homebridge Hue connects to the event stream.

Hi @ebaauw Thanks. I haven't set "forceHttp", so this should be fine. I can see via netstat that https is used.

But I cannot find any event stream messages when I press a Hue Dimmer button. Only the "button pressed" messages. And it feels not faster from the switching delay.

When should the event stream message appear in the log? On every action / press of a button or on the Homebridge start or anytime between periodically?

Thanks!

[ebaauw]

The firmware should be the newest available I think (1.48.1948086000)

I don't recognise that as a Hue bridge version, it seems to be some obscene combi of the api version and the firmware version. The latter part is not the latest firmware though, see below.

On startup, you should see:

[1/9/2022, 8:41:30 PM] [Hue] homebridge-hue v0.13.34, node v16.13.1, homebridge v1.3.9
(...)
[1/9/2022, 8:41:30 PM] [Hue] hue-3: Signify Netherlands B.V. BSB002 bridge v1949107040, api v1.48.0
(...)
[1/9/2022, 8:41:31 PM] [Hue] hue-3: event stream connected to https://hue-3/eventstream/clip/v2

When an event is received you should see something like (setting a light on using another API client):

[1/9/2022, 8:44:30 PM] [Hue] Living Room 9: state changed event: {"on":true}
[1/9/2022, 8:44:30 PM] [Hue] Living Room 9: light on changed from false to true
[1/9/2022, 8:44:30 PM] [Hue] Living Room 9: set homekit on from false to true

The first two are debug messages, the last is a regular log message. When pressing a dimmer switch:

[1/9/2022, 8:47:10 PM] [Hue] Hue dimmer switch 1: state changed event: {"buttonevent":1000,"lastupdated":"2022-01-09T19:47:10"}
[1/9/2022, 8:47:10 PM] [Hue] Hue dimmer switch 1: sensor buttonevent 1000 on 2021-12-21T13:17:16
[1/9/2022, 8:47:11 PM] [Hue] Hue dimmer switch 1: state changed event: {"buttonevent":1002,"lastupdated":"2022-01-09T19:47:11"}
[1/9/2022, 8:47:11 PM] [Hue] Hue dimmer switch 1: sensor buttonevent 1002 on 2022-01-09T19:47:10
[1/9/2022, 8:47:11 PM] [Hue] Hue dimmer switch 1 On: homekit button single press

Again, only the last message is a regular log message.

[jostrasser]

Is your Dimmer switch configured in Homekit or is it configured on the Hue Bridge / Hue App directly? In my case all Switches are configured in Apple Home to get the possibility to switch not only Hue lights (also other devices which are configured in Homebridge). If I configure the switches directly on the bridge then the response is much better.

Anyway: I can see these log entries. Looks fine. But I think it will not boost my switching speed. Right now I am having a delay of approx. 1-1,5 sec after pressing a dimmer button.

Thanks!

[ebaauw]

My Hue dimmer switches actually control the lights directly, without going through the bridge (or in my case: deCONZ gateway). Unfortunately, the Hue bridge doesn't support this configuration. Furthermore, I use a combination of bridge (deCONZ gateway) rules (for my Zigbee devices) and HomeKit automations (for my non-Zigbee devices). The bridge rules are faster and more reliable. Considering what's happening, a latency of 1s to 1.5s doesn't seem excessive:

• Zigbee device sending a notification to the Hue bridge;
• Hue bridge buffering events, sending them only once per second to Homebridge Hue (this is in fact the only delay, as in waiting time when no-one is processing);
• Homebridge hue, through Homebridge/HAP NodeJS sending a notification to the home hub;
• The home hub sending a command to the HomeKit accessory (typically another Homebridge plugin);
• The other homebridge plugin forwarding the command to the device (maybe even to another hub/bridge/gateway);
• The other hub/bridge/gateway forwarding the command to the device;
• The device executing the command.

versus

• Zigbee device sending a notification to the Hue bridge;
• Bridge sending a command to the other Zigbee device;
• Other Zigbee device executing the command.