dresden-elektronik / deconz-rest-plugin

deCONZ REST-API plugin to control ZigBee devices
BSD 3-Clause "New" or "Revised" License
1.88k stars 485 forks source link

IKEA Trådfri Remote (and motion sensor) #23

Closed jsve closed 5 years ago

jsve commented 7 years ago

IKEA has launched some more products in their Trådfri line. I have gotten the lights to work, but not remotes. They also have motion sensors now, which I guess will not work either.

de_web_plugin.cpp has a comment about IKEA remote not being supported yet. Is this because of the web-plugin of the core of deCONZ? Is there some work being put in to make this compatible?

I have managed to get the remote to show up in the deCONZ GUI and it has that blue indicator-dot that blinks when I push buttons. So something seems to be working at least. The logs (using --dbg-aps=2) shows the proper cluster being activated when i push the respective button, but there is nothing indicating if it was (for instance) brightness up or down that was pushed.

manup commented 7 years ago

Hi, in a few days we will release a new beta update which adds basic support for the Trådfri Remote in the REST API and realtime event notification in a new WebSocket interface. The button events can be used in rules to control lights and groups. We've made some tests with the Remote (firmware from 2016) and so far we haven't managed to find a stable way to join it into the gateway network — by pressing the setup button 4 times, sometimes it works, sometimes not. Hopefully the firmware will improve with the wider release.

We haven't testet the motion sensor yet, due the lack of a device, but expect support in the next few releases.

jsve commented 7 years ago

Awesome! My experiences with the remote is similar. Somehow it just pairs after a lot of tries. Not sure, but it seems to help if i spam the buttons shortly after the pair (or at the same time). Might just be a coincidence though.

Guess thats fine if you know about the issue. I think it would be OK if it was available in the web-gui as an option, but with the disclaimer that it might take a few tries.

Is there any way for me to test the beta when it comes out?

manup commented 7 years ago

Sure, you can download beta versions from https://www.dresden-elektronik.de/rpi/deconz/ the source code will be updated here too. Right now only stable version is kept here but we will provide a new branch (beta?) with the next beta release wich is in sync with the binary version.

Be aware that the next beta will only advance the REST API, not the WebApp itself. So using the IKEA switch and others is possible but a bit hacky.

manup commented 7 years ago

The beta is released now, with basic support for IKEA Trådfri motion sensor and remote control

https://github.com/dresden-elektronik/deconz-rest-plugin/releases

yene commented 7 years ago

Is it technically possible to talk directly to a light, or is always a remote required?

manup commented 7 years ago

Yes every light can be controlled directly through /lights REST API. The remotes are just a further way to control them.

binomie commented 7 years ago

Hi, I managed to join the remotes, but how do i join a light. After joining the remotes to raspbee they don't join the lights by long press any more. I have the Tradfri LED E27 1000 lm.

manup commented 7 years ago

The lights need to be brought into the network first. If they were paired to the remote before you need to reset them. In case of IKEA lights they can be resettet with 6x on/off powercycle the light.

Or you can reset lights under Menu > Settings and then Scan for Devices > Reset.

After that bring the lights into the network as usual with Open Network. Once they are in the network you can pair them with the remote or simply put them in the remotes group.

binomie commented 7 years ago

Many thanks, worked well. I always tried to pair with 10 seconds press like Ikea mentions in their description, now i understood to open Network and then reset the light (6x on off) or button (4x press on back button).

In sensor.cpp i saw only two ikea sensors. I have the dimmer as well:

IKEA Dimmer On Youtube

DeCONZ does show it on the map, but it does not get a name and is not shown in the webinterface. Is it not yet supported?

manup commented 7 years ago

Yes the integration into API will take some time and testing, the dimmer is more complicated to handle than for example the remote.

However since it's already in the network you can pair lights, which are also in the network, to it with the physical pairing button of the dimmer.

ebaauw commented 7 years ago

I couldn't resist and got myself a Trådfri light and remote. Both connect to deCONZ without any issues. The buttons on the remote cause WebSocket events, very nice! As far as I can tell, the value of /sensors/<id>/state/buttonevent matches the WebSocket notifications. I like that it uses similar buttonevent values to the Hue dimmer switch.

I do see some inconsistencies on the use of buttonevent, though:

  1. When pressing the On/Off button, a 1002 buttonevent is emitted. In Hue speak this would mean a release (after press) of the On button. A press would bebuttonevent 1000. When holding or releasing the On button, no additional events are emitted;
  2. Similarly, when pressing the Dim Up or Dim Down button, a 2002 or 2003 buttonevent is sent. I'm not sure if this is linked to the press or to the release (after press), though. When holding the Dim Up or Dim Down button, a single 2001 or 3001 buttonevent is emitted. In this case, the Hue dimmer switch would emit 2000 or 3000 (on press) followed by series of 2001 or 3001 while holding the button. When releasing (after hold) the Dim Up or Dim Down button, a 2003 or 3003 buttonevent is sent, just like the Hue dimmer switch.
  3. When releasing (after press) the Next (right) or Previous (left) button, a 5002 or 4002 buttonevent is emitted. Again, no equivalent 5000 or 4000. No event is emitted at all when pressing, holding, and releasing (after hold) these buttons. On the Hue dimmer switch, the 400x range is used for the Off button. The Hue dimmer switch has no equivalent for Next or Previous buttons.

While the current behaviour seems to match the Trådfri remote's functionality as a standalone dimmer, I would like to get the same series of buttonevents as for the Hue dimmer: x000 and x002 for press/release, and x000, x001, ..., x001, x003 for press, hold, ..., hold, release. I'm not sure if this is technical feasible, though: the remote doesn't seem to have an equivalent cluster to the Hue dimmer switch's FC00.

ebaauw commented 7 years ago

Somehow it just pairs after a lot of tries. Not sure, but it seems to help if i spam the buttons shortly after the pair (or at the same time).

@jsve I don't think this is a coincidence. I find reading attributes from the Trådfri remote as well as from the Hue dimmer switches in the deCONZ GUI usually fails. Based on a remark by @manup in another thread, on battery-powered devices being mostly asleep, I started hitting a button while reading the attributes... with increased success rate (but still not 100% by far).

Probably related to this: The config for the Trådfri remote doesn't show battery. Some of my dimmer switches don't show battery. I did a clean installs for all.

Hypfer commented 7 years ago

I have 6 Tradfri remotes. Adding the first one to the network did work well. However, all the other remotes just get hex values as their names and never show up as remotes.

Why is that?

Update:

It seems that the network wasn't completely open for adding new nodes. I have no idea why.

However pressing "Scan for devices" and then 5 seconds later "Open Network" followed by factory resetting the remote and mashing its buttons did work.

ebaauw commented 7 years ago

I put the Trådfri light in the Trådfri remote group and it reacts to On/Off, Dim Up, and Dim Down. It doesn't react to the Previous and Next buttons, thought. When paired directly, the light changes it's colour temperature on these buttons. I finally managed to create a binding of the Scenes cluster (from the remote to the light), but that doesn't seem to change anything.

philipptrenz commented 7 years ago

Short question. I'm on v2.04.35, pairing a Trådfri lamp worked fine. But I'm struggling with adding a remote. Do I have to upgrade? To which version, or even HEAD? Thanks!

manup commented 7 years ago

https://github.com/dresden-elektronik/deconz-rest-plugin/releases

The remote has early support in the latest beta release version 2.04.40, it creates a group to which you can add lights. Currently only on/off and dimming works.

ebaauw commented 7 years ago

One of my users has a 1000lm Trådfri bulb, which deCONZ (v2.04.35) reports as On/Off light, even though it's dimmable:

{
  "etag": "f8679f2533edfc067b383abbc7bf34bf",
  "hascolor": false,
  "manufacturer": "IKEA of Sweden",
  "modelid": "TRADFRI bulb E27 opal 1000lm",
  "name": "Decken-Lampe",
  "pointsymbol": {},
  "state": {
    "alert": "none",
    "bri": 0,
    "effect": "none",
    "on": false,
    "reachable": true
  },
  "swversion": "1.1.1.0-5.7.2.0",
  "type": "On/Off light",
  "uniqueid": "00:0b:57:ff:fe:xx:xx:xx-01",
  "manufacturername": "IKEA of Sweden"
}

I noticed the ZigBee device ID for a ZHA On/Off light is the same as that for a ZLL Dimmable light. I suspect our Swedish friends think they use the ZLL device ID, but that gets interpreted differently on the ZHA endpoint?

ebaauw commented 7 years ago

On google, I found the location of the Trådfri firmware: http://fw.ota.homesmart.ikea.net/feed/version_info.json. I downloaded the files for my bulb and remote and renamed them to end with .zigbee, but the deCONZ OTAU plugin doesn't seem to recognise them. Is it at all possible to use deCONZ to upgrade the firmware of non-dresden-elektronik devices?

manup commented 7 years ago

I noticed the ZigBee device ID for a ZHA On/Off light is the same as that for a ZLL Dimmable light. I suspect our Swedish friends think they use the ZLL device ID, but that gets interpreted differently on the ZHA endpoint?

This is a HA/ZLL glitch in the ZigBee spec, I think they clarified that in the ZigBee 3.0 spec.

The new commit https://github.com/dresden-elektronik/deconz-rest-plugin/commit/f0f3e95c4083d0408acdc2e4c0724421d8b2a35a fixes the issue, IKEA bulbs now have proper type 'Dimmable light'.

Is it at all possible to use deCONZ to upgrade the firmware of non-dresden-elektronik devices?

Yes, the plugin just uses standard ZigBee OTA, It was already testet with OSRAM, Busch-Jaeger and JUNG/GIRA devices. However the OTA firmware files of these are not publicly available (yet).

The firmware files of IKEA are raw data, not packet into an standard OTA file format. So in order to use them the .zigbee format binary header must be put before the data payload as described in the spec. Right now it's not worth the efford, because the version referenced in the JSON file is only the 'old' shipped version (1.1.1.0-5.7.2.0). I guess you're looking for the new ZLL certified version ;-) to my knowledge it's not online yet.

Once it's online (and downloadable like the current version) I'll give it a go for an update test.

ebaauw commented 7 years ago

The remote has early support in the latest beta release version 2.04.40, it creates a group to which you can add lights. Currently only on/off and dimming works.

Still some issues in v2.04.46:

perler commented 7 years ago

Ok, I am still a bit unclear. I can bind the Tradfri remote to the deConz gateway in a way that it appears in the app but not in the webui. How can I control devices now?

manup commented 7 years ago

Normally it should create a new group to which you can add lights in the webapp.

perler commented 7 years ago

it didn't.. but I added the switch in 2.0.35 (when support was not really there for it), is there a way to force/delete it first?

manup commented 7 years ago

Please try the following with a recent deCONZ version:

It's a bit hacky but now a new sensor should be created in the API and the group should also be created.

perler commented 7 years ago

that didn't work unfortunately. I really would like to remove the device from deconz first. is there a db where I can simply delete it?

manup commented 7 years ago

Yes it's a SQLite database

.local/share/data/dresden-elektronik/deCONZ/zll.db

It can be edited with tools like sqliteman, deleting the file is also possible but this would discard lights and groups settings as well. The switch if present is listed under sensors.

While modifying the database deCONZ shall not run.

perler commented 7 years ago

I did a complete factory reset and this looks more and more like a bug to me.

the tradfri remote is visible in the GUI (it blinks blue (in the GUI)) when a battery is inserted but it is nowhere to be seen in the web gui and so no group is created and controlling devices with the remote is not possible.

manup commented 7 years ago

Can you please try out the new version 2.04.52, it should improve the discovery of the remote.

http://www.dresden-elektronik.de/rpi/deconz/beta/deconz-2.04.52-qt5.deb

Further if this worked so far, please press the large button again for more than 5 seconds to finish setup for the arrow buttons (control color temperature).

ebaauw commented 7 years ago

Deleted the Trådfri remote resource through the API, deleted the node it from the deCONZ GUI, deleted the record (with status deleted) from the database, installed 2.04.52.

Opened the network through the Webapp, pressed the On/Off button. The Trådfri remote appears as a node in deCONZ GUI. Another press or two and sensor resource is created. Read the attributes of the Basic and Power Configuration clusters in the deCONZ GUI, just to be sure. A group was created, and linked to the remote (through the group's devicemembership attribute and through the sensor's config.group attribute). I got web socket notifications for the creation of the sensor and for the buttonevents (1002, 2002, 2001/2003, 3002, 3001/3002, 4002, 5002), as expected.

Added the Trådfri bulb to the group. The light now changes colour temperature on the Previous and Next buttons! When changing the colour temperature, I even get two web socket notifications (as expected): one for the sensor state.buttonevent and one for the light state.ct.

Still some issues, though:

manup commented 7 years ago

Wow that was fast, thanks for testing :)

  • The Trådfri remote still doesn't show under Devices in the Webapp;

Not in the current public one, we are very busy working on a complete new app (in development for about a year now) which includes several switches and sensors, which are currently only visible in the API. There will be a private beta soon to test these things out.

  • The light (and group) state.on and state.bri aren't updated on the On/Off, DimUp, or DimDown buttons;

This is a general problem related to specific ZigBee commands of various remotes/switches which I want to address in the next days. The events for groups should also be improved by this.

  • The light state.ct reflects the value sent to the light, not the actual value of the light (e.g. it reports ct in the range of 153..500, where the Trådfri only supports ct of 250..454). When the light is polled, I receive a new event for the correct state.ct value.

Yep currently its the hard coded 153..500 range (originated by first E27 hue bulbs), this will be changed to reflect the actual physical min. max. values reported by the lights.

  • config.battery is still missing from the sensor resource.

Currently disabled and needs more testing, not sure if the remote supports ZCL attribute reporting for power configuration cluster as the hue dimmer switch does.

It's a quite fuzzy problem since many battery powered devices provide very different ways to report battery state, or even only battery critical events.

  • Probably more a caveat than an issue: The Previous / Next buttons react differently from when the remote is linked directly to the light: in that case, Ikea only support three values for ct and the value rotates (e.g. pressing Next on the max value switches over to the min value).

That was made on purpose and is considered a feature :) since the original IKEA way only provides the 3 temperature settings, we decided it would be nicer to have finer smooth control as for brightness. The step size and fading time needs some tweaking though.

Later on we will provide an alternative mode to cycle through all scenes of the underlying group.

ebaauw commented 7 years ago
  • Probably more a caveat than an issue: The Previous / Next buttons react differently from when the remote is linked directly to the light: in that case, Ikea only support three values for ct and the value rotates (e.g. pressing Next on the max value switches over to the min value).

That was made on purpose and is considered a feature :) since the original IKEA way only provides the 3 temperature settings, we decided it would be nicer to have finer smooth control as for brightness. The step size and fading time needs some tweaking though.

I see you created rules for the Previous and Next buttons, instead of using a direct binding between the Trådfri Remote and the group. I understand that this is needed for non-Trådfri bulbs. However, now when deCONZ is not running, the On/Off, DimUp, and DimDown buttons continue to work, but the Previous and Next buttons don't.

I'm currently migrating my entire setup, Hue lights, Hue motion sensors, Hue dimmer switches, CLIP sensors and rules from the Hue bridge to the deCONZ gateway. Looking good so far: deCONZ seems to like my custom-made rules, and the CLIP sensors seem to work just fine now.
Unlike the Hue bridge where you either bind the switch to the bridge or to a group, deCONZ seems to support both bindings simultaneously. I really like that with deCONZ, the dimmer switches continue to work when the gateway is down, still allowing for more advanced use in rules when the gateway is running.
Could you elaborate how that's done? Is the gateway a "member" of the switch group (listening in on the commands send to the group)? Or are there multiple bindings, from the 0xFC00 cluster to the gateway and from the 0x0006/0x0008 clusters to the group? How is that done for the Trådfri Remote, which doesn't have a 0xFC00 cluster?
What is the mode attribute of the sensor? It's 3 for the Trådfri remote and 1 for the Hue dimmer switches. It's not reported for other sensor resources, but in the database they have value 1 as well. I've also seen value 2 in the database (I think after creating a CLIP sensor, but before setting its state?).

I tried changing group IDs in the database (both in the nodes, sensors, and groups tables), but deCONZ insisted on using its own group IDs with the Hue dimmer switches and Trådfri remote. Changing sensor IDs and node IDs worked out just as I expected. (I'm using a numbering scheme for the IDs, so I can more easily maintain my rules manually).

It would be cool to have a way to create, view, and maintain the bindings from the REST API (also for other sensors like the Hue motion sensor). I've seen the documentation for the BIND rules, but no such rules are created automatically. Also their syntax is awful. Would it be possible to create a /bindings resource instead, or alternatively, create a config.bindings array for the sensor, which would contain and object with the (sensor) cluster and (group/gateway) endpoint for each binding?

perler commented 7 years ago

I still can't get the remote visible in the web app. I tried everything as described in this thread. I can see it in deConz, but not on the web app. commit 545f771f7b3654ecefa2a14e71430c7c2f6d6c8c

ebaauw commented 7 years ago

The trådfri remote doesn't show in the webapp, but you should see the group that was created for it. It does show in the API under /sensors.

manup commented 7 years ago

I still can't get the remote visible in the web app. I tried everything as described in this thread. I can see it in deConz, but not on the web app. commit 545f771

Did you try it with version deCONZ 2.04.53? There were some fixes after 2.04.52 to improve the setup. We've seen different behaviors from IKEA remotes related to the large button.

Can you also check which firmware version is running on the RaspBee/ConBee: In deCONZ > Help > About the firmware version should be 0x26110500.

perler commented 7 years ago

the commit mentioned is 2.04.53..

but the firmware update did the trick. I have seen the "update available" information in the web app but thought it was a bug (as I was on the latest commit). I was not aware that this is about the firmware.

manup commented 7 years ago

I see you created rules for the Previous and Next buttons, instead of using a direct binding between the Trådfri Remote and the group. I understand that this is needed for non-Trådfri bulbs. However, now when deCONZ is not running, the On/Off, DimUp, and DimDown buttons continue to work, but the Previous and Next buttons don't.

Currently that's the only way to use the buttons with deCONZ, since the remote only sends a fixed set of commands on button press to the related group, which can't be changed. Maybe the IKEA gateway is able to configure the buttons but I haven't looked deeper into this yet, would be interesting though :)

I really like that with deCONZ, the dimmer switches continue to work when the gateway is down, still allowing for more advanced use in rules when the gateway is running. Could you elaborate how that's done? Is the gateway a "member" of the switch group (listening in on the commands send to the group)? Or are there multiple bindings, from the 0xFC00 cluster to the gateway and from the 0x0006/0x0008 clusters to the group? How is that done for the Trådfri Remote, which doesn't have a 0xFC00 cluster?

It's a mix: the hue dimmer switch is quite unique since binding could be created to customize it to some extend, Busch-Jaeger switches also support this freedom - but it's more tricks to setup. Most other switches like IKEA, JUNG, GIRA and ours create their own random 16-bit group to which lights can be added. They just send commands (on/off, dimm, call scene x, ...) to the related group and don't (need to) care what happens then. Setup for these is quite simple.

The gateway receives all group commands in the network too (in newer firmware versions). And is able to extract the related group-id and specific command which was send.

Adding lights to the switch group and edit scenes for certain buttons is quite easy via the API and our new App is using this quite heavily and if possible always takes advantage of the switch group so features will work even when the gateway is off.

manup commented 7 years ago

but the firmware update did the trick. I have seen the "update available" information in the web app but thought it was a bug (as I was on the latest commit). I was not aware that this is about the firmware.

Not a bug but poorly implemented user experience :)

The two updates will be combined in near future, we've seen confusion about it quite often.

ebaauw commented 7 years ago

but the firmware update did the trick

Do you see the Trådfri Remote under Devices in the web app? I'm on deCONZ v2.04.53, firmware 0x26110500 (according to deCONZ gui's About). I deleted the remote and corresponding group, restarted deCONZ, opened the gateway and reset the remote (4x press the reset/link button). deCONZ found the remote, created the sensor resource and the group, but it's not visible in the web app.

perler commented 7 years ago

yes, right now I see it but didn't find the time to use it for anything.

manup commented 7 years ago

deCONZ found the remote, created the sensor resource and the group, but it's not visible in the web app.

As mentioned before in the current web app it won't show up as a device only the group and in the REST API. Switches and Sensors like the IKEA ones are only implemented in the new app which is not yet public. By interest I can provide early access to it next week.

perler commented 7 years ago

you are right, I see only the group. I would be interested in the beta..

ebaauw commented 7 years ago

By interest I can provide early access to it next week.

Yes, please.

I did notice a new Daylight Control option under Settings, which found the light level resources for my Hue motion sensors. Nice! However, it does show the gateway time in PM, even tough I set the time format to 24h.

Most other switches like IKEA, JUNG, GIRA and ours create their own random 16-bit group to which lights can be added

Dare I ask: Ubisys?

Where is this group stored (which cluster/attribute on the switch)? I cannot seem to find it in the deCONZ GUI.

Currently that's the only way to use the buttons with deCONZ, since the remote only sends a fixed set of commands on button press to the related group, which can't be changed.

I appreciate that, but why doesn't this command reach the Trådfri bulb in the remote's group?

donnib commented 7 years ago

Can one use the 5 buttons on the IKEA remote to control anything you like ? I mean if i just get a IKEa remote can i use it for something else than together with the bulbs ? I pressume the answer is yes since the remote is paired with the RaspBee gateway ?

ebaauw commented 7 years ago

Can one use the 5 buttons on the IKEA remote to control anything you like ?

Yes, they're available as buttonevent values to the sensor resource, so you can use them in gateway rules, controlling anything connected to the deCONZ gateway. The functionality is different compared to the Hue dimmer switch, see above.

Using my homebridge-hue plugin, you can use them in HomeKit automations as well, controlling anything connected to HomeKit.

donnib commented 7 years ago

@ebaauw nice, i am just about to get my hand on a bunch of bulbs and ikea remotes brand new at 50% discount so nice to know i can use them as i want. It seems like the motion sensor is the same case right ? I am hoping to use this with Home Assistant which i like a lot and got the hang of it but i have no idea if the websockets events work with that....hmm. Have you tried these switches http://www.ikea.com/dk/da/catalog/products/00356910/ ? I also have a Hue motion sensor which i hate how i have to poll to the gateway. I have ordreded an RaspBee and hope i can use that for Philips, Ikea and my Osram lights.

ebaauw commented 7 years ago

I have no experience with the Ikea motion sensors, but they should work. I use (many) Hue motion sensors. I also have a bunch of Hue dimmer switches, two Hue taps, and one Ikea Trådfri remote. All these fire websocket events in the latest beta (I'm on v2.04.54) as one would expect. I have yet to make use of use these events in my homebridge plugin, though.

I don't have any Trådfri dimmers, afaik they don't work with deCONZ.

I have Philips, OSRAM, innr, and Ikea lights connected to deCONZ. Also an OSRAM plug and a ubisys dimmer.

donnib commented 7 years ago

Sound very good, now i need to find out if Home Assistant can support Websockets in it's Hue components (seems this works with the RaspBee gateway).

@manup do you know anybody using Home Assistant with RaspBee ?

neonal commented 7 years ago

@donnib I am, with a bit of a problem: https://github.com/dresden-elektronik/deconz-rest-plugin/issues/66

ebaauw commented 7 years ago

If I read the ZigBee sniffing data correctly, the Trådfri remote does send different commands when pressing (0x07), holding (0x08) and releasing (0x09) the Previous and Next buttons, with different data to indicate which of the buttons was pressed. This means that, technically, we could support button events 4001, 4003, 5001, and 5003 in addition to 4002 and 5002 for the Remote.

donnib commented 7 years ago

👍 for battery support on the IKEA Remote, that would be great so i can monitor those.

@manup If i use deConz together with my IKEA devices, how can i update the firmware ? Can i download it and push it thru deConz ?