Warning: breaking changes in homebridge-hue v0.8

ebaauw commented 6 years ago

homebridge-hue v0.8 will break your existing HomeKit setup: HomeKit will no longer recognise the accessories and services as the ones previously exposed by an older version of homebridge-hue. This will cause the accessories and services to be dropped from any HomeKit room, group, scene, and automation. Effectively, you'll have to redo you HomeKit setup.

I don't particularly enjoy this myself: it typically takes me half a day to redo my HomeKit setup. However, I've been putting off these changes for far too long, and I need these for further development of homebridge-hue. This has, in fact, been a major impediment for moving towards dynamic platform accessories (issue #4).

When I started homebridge-hue, several years ago, the world of Hue was simple. The Hue bridge exposed a ZigBee device as a single REST API resource, which homebridge-hue exposed as a single HomeKit HomeKit service, for which homebridge created the corresponding HomeKit accessory. With the introduction of the Hue motion sensor, but even earlier with e.g. the dresden elektronik wireless ballasts, this 1:1 relationship between device, resource, service, and accessory no longer holds. The Hue motion sensor is one device, exposed as three /sensors resources by the REST API (ZLLPresence, ZLLLightLevel, ZLLTemperature), which homebridge-hue actually exposes as five different HomeKit services (Motion Sensor, Ambient Light Level Sensor, Temperature Sensor, Battery and Accessory Information), combined in a single HomeKit accessory. For stateless switches the setup is even more complicated (with a HomeKit service per button). More recently, fakegato-history has added an additional service for the history in Eve.

Until now, I've been able to handle these changes with the original design of homebridge-hue: expose each resource as a separate HomeKit service, and afterwards glue them together into a single HomeKit accessory. However, the current pre-v0.8 design of homebridge-hue still doesn't recognise or manage HomeKit accessories. Over time, the code to workaround this limitation has become almost too complex to maintain. Furthermore, the current design has serious limitations:

Obviously, to support dynamic platform accessories (issue #4), I need to manage the accessories;
For smart plugs, I need to combine multiple resources into one HomeKit service (one /lights resource to control the plug, and a ZHAConsumption and/or ZHAPower /sensors resource for metering). For that, I need to know the accessory upfront;
For weather sensors, the order of the ZHATemperature, ZHAHumidity and ZHAPressure /sensors resources is relevant: the ZHAHumidity needs to be first or second or the history service won't be created;
I cannot expose history for a temperature sensor (without humidity), as this might interfere with the motion history of the Hue motion sensor, again depending on the order of the /sensors resources;
For chandeliers/luminaries consisting of multiple lights, I want to combine multiple /lights resources into one HomeKit accessory (in my never ending and ultimately doomed quest to avoid the dreaded 99 accessory limit);
I need to whitelist explicitly each light with multiple /lights resources (see #198).

Starting with v0.8, homebridge-hue will manage accessories explicitly. The discovery of bridge resources has been refactored, to collect resources into HomeKit accessories first, and second to expose the resources as HomeKit services in the right order and in context of the HomeKit accessory (rather than first exposing each resource as a HomeKit service, in the (random) order as provided by the REST API, and second glueing these services into a HomeKit accessories). However, as a consequence of this, the IDs (and derived UUIDs) of the HomeKit accessories and services change, causing HomeKit to see new accessories instead of existing ones.

DSchumacher2104 commented 6 years ago

Can you tell, when v0.8 will be released? My homebridge crashed yesterday, so i need to a rebuild my config anyway

ebaauw commented 6 years ago

I hope later today. Currently testing against my production deCONZ. Still some funny stuff around smart plug history...

DSchumacher2104 commented 6 years ago

Okay, then it‘s better to wait.

ebaauw commented 6 years ago

v0.8.1 is out. There's still some work to do (e.g. blinking the lights on Identify), but I don't expect any more breaking changes. If you are planning a re-installation, this is the moment to upgrade, otherwise you might want to delay a bit.

iantait commented 6 years ago

I've installed it - on startup I get a catch:

[2018-3-2 18:19:17] [Hue] Philips hue: TypeError: Cannot read property 'match' of undefined at HueBridge.getSerialNumber (/usr/local/lib/node_modules/homebridge-hue/lib/HueBridge.js:107:26) at HueBridge.exposeSensor (/usr/local/lib/node_modules/homebridge-hue/lib/HueBridge.js:491:29) at HueBridge.exposeSensors (/usr/local/lib/node_modules/homebridge-hue/lib/HueBridge.js:484:14) at HueBridge.exposeResources (/usr/local/lib/node_modules/homebridge-hue/lib/HueBridge.js:421:8) at getConfig.then.then.then.then (/usr/local/lib/node_modules/homebridge-hue/lib/HueBridge.js:92:17) at Function.then (/usr/local/lib/node_modules/homebridge-hue/node_modules/deferred/_ext.js:67:13) at Deferred._settle (/usr/local/lib/node_modules/homebridge-hue/node_modules/deferred/deferred.js:97:26) at Deferred.resolve (/usr/local/lib/node_modules/homebridge-hue/node_modules/deferred/deferred.js:142:15) at request.then (/usr/local/lib/node_modules/homebridge-hue/lib/HueBridge.js:316:11) at Function.then (/usr/local/lib/node_modules/homebridge-hue/node_modules/deferred/_ext.js:67:13)

iantait commented 6 years ago

I still get the catch with 0.8.1

iantait commented 6 years ago

I've got no stuff attached to the hue bridge anymore btw.

iantait commented 6 years ago

Everything else looks good though. Should I be getting history for the hue motion sensor temperature now?

ebaauw commented 6 years ago

[2018-3-2 18:19:17] [Hue] Philips hue: TypeError: Cannot read property 'match' of undefined I've got no stuff attached to the hue bridge anymore btw.

Damn. I thought I'd tested everything, but apparently not the Hue bridge built-in Daylight sensor, which doesn't have a uniqueid. Will be fixed shortly. You could workaround it by specifying "excludeSensorTypes": ["Daylight"] in config.json.

Everything else looks good though.

Thanks, glad to hear that!

Should I be getting history for the hue motion sensor temperature now?

No, not yet. I would need to expose the Temperature sensor as a separate accessory. There's only one History service per accessory. Elgato doesn't have a device which combines Motion and Temperature, so we don't know what the history format would need to look like, if it even would be possible to combine them. I did lay the groundwork for doing that later, though (probably using a config.json flag, as I'm struggling with HomeKit's 99 bridged accessories limit).

iantait commented 6 years ago

All good stuff, thanks!

Somberland commented 6 years ago

Looking forward to this. Should I use the homekit feature of the hue bridge or should I add the bride to homebridge and use homekit via homebridge combined with deconz etc?

ebaauw commented 6 years ago

Should I use the homekit feature of the hue bridge or should I add the bride to homebridge and use homekit via homebridge combined with deconz etc?

homebridge-hue works with deCONZ and the Hue bridge simultaneously. It also works in combination with the native HomeKit feature of the Hue bridge.

If you want the homebridge-hue features, like history in Eve for the Hue motion sensor, non-Hue lights, long presses on the Hue dimmer switch, you better expose the Hue bridge through homebridge-hue. Note that hombridge-hue needs to poll the Hue bridge, so exposing the Hue sensors and switches natively might give better response times.

You could also consider moving all your devices to deCONZ and turn off your Hue bridge. That would give you push notifications and advanced features. Keep the bridge, though, for any future firmware updates of Hue lights, sensors, and switches.

to0b commented 6 years ago

Hey Erik,

I've created a CLIPGenericFlag on my hue Bridge, but with 0.8.1 it will not exposed to HomeKit. The deCONZ CLIPGenericFlag was successfully exposed. Can you please have a look at this feature?

Edit: Ok it was the same error @iantait described. Exclude the Daylight Sensors worked.

marioaltvater commented 6 years ago

I got this error at first, but after restarting deCONZ it started to work again.

[2018-3-3 11:56:22] [Hue] RaspBee-GW: 2 services
[2018-3-3 11:56:22] [Hue] Error: Cannot add a bridged Accessory with the same UUID as another bridged Accessory: 7cc0b286-3a03-4428-9287-54d07a929d72

No problems so far. Thanks for the update. 🙂

Razeeer commented 6 years ago

Hey Erik,

after the Update I get this warning message at starting the Homebridge:

Huebridge Abs: /lights/18: warning: using default colour gamut for unknown light model {"state":{"on":false,"bri":254,"hue":38912,"sat":20,"effect":"none","xy":[0.326,0.3253],"alert":"none","colormode":"xy","mode":"homeautomation","reachable":false},"swupdate":{"state":"notupdatable","lastinstall":null},"type":"Color light","name":"Lampe Balkon","modelid":"Gardenspot RGB","manufacturername":"OSRAM","capabilities":{"certified":false,"streaming":{"renderer":false,"proxy":false}},"uniqueid":"84:18:26:xx:xx:xx:xx:xx-03","swversion":"V1.04.12"}

ebaauw commented 6 years ago

@Razeeer, yes, my bad - will be fixed i v0.8.2, see #300.

@marioaltvater, what did you change between "at first" and "after restarting"? Do you only have one deCONZ gateway, or did you recently more devices from a Hue bridge or deCONZ?

@to0b, OK - I just double checked: a CLIPGenericFlag from the Hue bridge is exposed normally.

@iantait, homebridge-hue will actually already create history for the temperature sensor of the Hue motion sensor... if you disable the ZHAPresence sensor. The quickest way to get both histories is to use two homebridge instances. Interestingly, I changed my test config without deleting the history file or re-installing Eve. Eve now shows 0,00˚C entries for the old motion history. However the temperature log no longer shows one entry per 10 minutes - every motion event is reflected. Fundamentally, the history for motion (and door) sensors works differently (event based) from the history for weather and energy sensors (periodic).

ebaauw commented 6 years ago

Just released v0.8.2, see release notes.

Still to do:

Fix lights not blinking on identify;
History for both motion and temperature sensors in Hue motion sensor;
Multi-light accessories (cf. multi-CLIP), combining multiple lights into one accessory.

stefan1957 commented 6 years ago

Hey Erik, I have got a view problems in Homebridge Hue and one of them is this one. What do i need to do need to solve it? [DeConz] deCONZ-GW: /sensors/19: warning: unknown ZHAPresence sensor {"config":{"duration":60,"on":true,"reachable":true},"ep":1,"etag":"9a0e587cb0ee2f722a546cee5ae43982","manufacturername":"LUMI","modelid":"lumi.sensor_wleak.aq1","name":"Lekkage","state":{"lastupdated":"2018-02-23T06:36:27","lowbattery":false,"presence":false,"tampered":false,"temperature":2300,"voltage":3025},"type":"ZHAPresence","uniqueid":"00:15:8d:00:01:bb:88:c5-01-0500"} Thx in advance Grtz Stefan

stefan1957 commented 6 years ago

Hey Erik,

The other problems is very strange. When i use version 0.7.2 everything in Homekit is working fine but when i switch to 0.8.2 the automation in homekit isn't working anymore everything else is working fine. In the picture is made with 0.7.2 and when i bump to 0.8.2 and create the same sensors nothing works anymore and they switch to off?? I hope you have a solution for that. Thx in advance Grtz Stefan fullsizeoutput_895

Nastras commented 6 years ago

Hi Erik, i have received this message since the update to 0.8.2 Can you do something with that?

HAP Warning: Characteristic E863F10A-079E-48FF-8F27-9C2605A29F52 not in required or optional characteristics for service 00000080-0000-1000-8000-0026BB765291. Adding anyway.`

marioaltvater commented 6 years ago

@ebaauw - I am using the deCONZ Raspberry Pi Image which had the X11 version of deCONZ in the autostart. With only one deCONZ gateway. Also after my comment from before, I kept running into the mentioned error and had to restart deCONZ manually to get rid of it after every start of homebridge. I have changed that now and I am only using the deCONZ service headless. (sudo systemctl enable deconz) The error disappeared.

ebaauw commented 6 years ago

[DeConz] deCONZ-GW: /sensors/19: warning: unknown ZHAPresence sensor {
  "config": {
    "duration": 60,
    "on": true,
    "reachable": true
  },
  "ep": 1,
  "etag": "9a0e587cb0ee2f722a546cee5ae43982",
  "manufacturername": "LUMI",
  "modelid": "lumi.sensor_wleak.aq1",
  "name": "Lekkage",
  "state": {
    "lastupdated": "2018-02-23T06:36:27",
    "lowbattery": false,
    "presence": false,
    "tampered": false,
    "temperature": 2300,
    "voltage": 3025
  },
  "type": "ZHAPresence",
  "uniqueid": "00:15:8d:00:01:bb:88:c5-01-0500"
}

@stefan1957, this is still the old sensor resource, as created by deCONZ a couple of versions ago. Please delete the sensor resource (ph delete /sensors/19) and re-pair the leak sensor. deCONZ should create a ZHAWater sensor, which is recognised by homebridge-hue.

ebaauw commented 6 years ago

when i switch to 0.8.2 the automation in homekit isn't working anymore everything else is working fine

@stefan1957, when upgrading, your automations become invalid, because the sensors in them no longer exist (HomeKit sees "new" sensors because of the changed UUID). You could try and re-select the sensor, but probably best delete the automation and create a new one from scratch.

create the same sensors

Not sure what you mean here.

stefan1957 commented 6 years ago

Erik, When i bumped the home bridge hue to version 0.8.2 I started all over with home kit, with a new house, and from that moment the automation does not work anymore. I am back to 0.7.1 and the automation is working fine now? I hope you understand what i mean Grtz Stefan

ebaauw commented 6 years ago

HAP Warning: Characteristic E863F10A-079E-48FF-8F27-9C2605A29F52 not in required or optional characteristics for service 00000080-0000-1000-8000-0026BB765291. Adding anyway.

@Nastras , this warning can be igored. homebridge-hue is trying to access a not-yet added Voltage characteristic in a Contact Sensor service. deCONZ v2.05.09 reports the battery voltage of Xiaomi devices as state.voltage, but state.voltage only appears after the first report from the sensor. homebridge-hue only configures the characteristics at startup, when state.voltage isn't there yet.

The next version of deCONZ will expose config.battery instead of state.voltage. Also config.battery which will be added to the resource on creation (or restore from the database).

EDIT That's 2.05.10, which just came out.

ebaauw commented 6 years ago

When i bumped the home bridge hue to version 0.8.2 I started all over with home kit, with a new house, and from that moment the automation does not work anymore. I am back to 0.7.1 and the automation is working fine now? I hope you understand what i mean

I don't know what you mean by "create the same sensors".

When you start a new home, your home hub (Apple TV, HomePod or iPad) needs to pick it up, before automations work. In the Home app, in the Home tab touch the arrow in the left upper corner to check that your home hub is connected.

There's nothing homebridge, homebridge-hue, or any HomeKit device can do to break or make HomeKit automation. I would suspect that the time you took to revert to 0.7.1 was just enough for the home hub to connect to the new home.

ebaauw commented 6 years ago

@marioaltvater by the sounds of it, I would guess homebridge was starting too soon, before deCONZ fully initialised all the resources in the REST API. When starting homebridge automatically on boot, you might want to create a lightlist to make homebridge-hue wait for deCONZ to have discovered all lights, see #246. Running deCONZ headless might speed up deCONZ initialisation, causing the timing issue to be resolved.

Should you run into this problem again, please open a new issue and attach the full debug log (see https://github.com/ebaauw/homebridge-hue#debug-log-file) and the debug dump file (see https://github.com/ebaauw/homebridge-hue#debug-dump-file).

stefan1957 commented 6 years ago

i tested this morning with version 0.8.2 a few hours to get the automation working but it failed. I can created one but it don't work. for example: If i want to turn the light in the garden on when open my kitchen door i can created it but it don't work and is turned off by homekit. If i turned it on again homekit turned it off and so on. Homekit sees that the door went open but the light stays off I hope you understand it better now. Gr Stefan

ebaauw commented 6 years ago

@stefan1957, yes this is very frustrating, but has nothing to do with homebridge-hue. HomeKit is syncing the automations between your various devices over iCloud, and the changes you just made get reverted. See also https://github.com/ebaauw/homebridge-hue/issues/118.

I find it helps, when doing major HomeKit setup changes, to turn off all iDevices, AppleTV, etc connected to the same iCloud account, except one iPad temporarily setup as home hub. Then create all the scenes, automations, and other settings (like order of accessories, which ones are favourite, which ones do notifications, etc) on that iPad. After everything works, power on the other devices one by one, waiting until the new config has been synced to the device, before powering up the next device.

ebaauw commented 6 years ago

homebridge-hue v0.8.3 and deCONZ v2.05.10 are out.

Please open a new issue if you find a new problem.

stefan1957 commented 6 years ago

Thx Erik, It was indeed the problem with the other devices and iCloud, everything is working fine now Grtz Stefan

Vetti52 commented 6 years ago

Hi! being pretty new to homebridge and deConz, I still succeeded to let it work up to this point: Homebridge works fine with the FritzBox plugin, but the hue plugin, using only deCONZ-GW, is presented as being "not supported" on myHome (iPad). I use homebridge-hue v0.8.3, node v8.10.0, homebridge v0.4.38, deCONZ v.2.04.99 (updated to deconz-latest.deb). My RaspBee is linked to a Lightify plug, which is controlled by the deCONZ WebApp, but still not by Homebridge Hue. Homebridge and deCONZ are started during startup, not via X11. No errors, only tho warnings, of course avahi, "not using recommended node version v8.9.4 LTS", "not using recommended deCONZ gateway version 2.5.10" . output of systemctl status deconz: ● deconz.service - deCONZ: ZigBee gateway -- REST API Loaded: loaded (/etc/systemd/system/deconz.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2018-03-08 21:48:12 CET; 16h ago Main PID: 370 (deCONZ) CGroup: /system.slice/deconz.service └─370 /usr/bin/deCONZ -platform minimal --http-port=80 raspberrypi systemd[1]: Started deCONZ: ZigBee gateway -- REST API. raspberrypi deCONZ[370]: libpng warning: iCCP: known incorrect sRGB profile raspberrypi deCONZ[370]: This plugin does not support propagateSizeHints() last line repeated two times.

If this is a new issue, please move it there (it's my first comment here), sorry. Anyway, any help is appreciated. Thank you Peter

ebaauw commented 6 years ago

If this is a new issue

It's not releated to this topic, but definitely not new. Please read the README, in particular https://github.com/ebaauw/homebridge-hue#configuration and https://github.com/ebaauw/homebridge-hue#prerequisites.

Vetti52 commented 6 years ago

Thanks, ebaauw, for your reply.

Of course I had read the homebridge-hue README, otherwise I would not have succeeded so far. It is really essential!!! Going through the prerequisites, I can only see one mistake in my assumptions, as I thought, that the „latest“ deCONZ deb version would be ok. I did not check, that it seems to be mandatory to install a beta version „later“ than the latest. I will use this weekend for trying that. Concerning the configuration part, I am not sure, what you mean. As I don‘t have either a round or square hue bridge version, but only a Raspber with deCONZ, I used none of examples, mentioned in the configuration section, but the config.json, which was produced when unlocking the deCONZ gateway containing the api keys. Additionally, the only thing I inserted, was a „host“ : „“ parameter, after which the plugin started without errors. Except, that it is still „not supported“ by Homekit. So, I was pretty sure, everthing works fine. Maybe I missed another README, where to get a functional example for a pure deCONZ config.json without any hue bridge involved. So please tell me.... But first of all I will update to the beta version of deCONZ, which is obviously mandatory.

Thanks in advance Peter

ebaauw commented 6 years ago

Yes, best use the latest beta of deCONZ, v2.05.11. You need another app than Apple's Home to use the gateway accessory. You need to specify in config.json what you want homebridge-hue to expose: lights, sensors, etc, otherwise you'll only see the gateway accessory.

Vetti52 commented 6 years ago

Thanks, Erik!Yes, it works fine now, even with iPAD Home App! Nice weekendPeter--------- Original Message --------- From: Erik Baauw To: ebaauw/homebridge-hue Date: Fri Mar 09 22:56:36 GMT+01:00 2018 Subject: Re: [ebaauw/homebridge-hue] Warning: breaking changes in homebridge-hue v0.8 (#299)

Yes, best use the latest beta of deCONZ, v2.05.11. You need another app than Apple's Home to use the gateway accessory. You need to specify in config.json what you want homebridge-hue to expose: lights, sensors, etc, otherwise you'll only see the gateway accessory.

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/ebaauw/homebridge-hue","title":"ebaauw/homebridge-hue","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/ebaauw/homebridge-hue"}},"updates":{"snippets":[{"icon":"PERSON","message":"@ebaauw in #299: Yes, best use the latest beta of deCONZ, v2.05.11. You need another app than Apple's Home to use the gateway accessory. You need to specify in config.json what you want homebridge-hue to expose: lights, sensors, etc, otherwise you'll only see the gateway accessory."}],"action":{"name":"View Issue","url":"https://github.com/ebaauw/homebridge-hue/issues/299#issuecomment-371956457"}}}

ebaauw / homebridge-hue

Warning: breaking changes in homebridge-hue v0.8 #299