Integrate with Ikea Trådfri Gateway as bridge to communicate and control connected Ikea's ZigBee based smart lights, switches, and sensors

[Hedda]

Ikea have just released a new app-controlled network-attached home automation hub which will serve as a Gateway to control its new "Trådfri" series of affordable smart lights / lightbulbs, switches / remotes, and sensors, which in turn so far all uses ZigBee based protocols. These products are set to be released on the 31st of March 2017 in selected countries around the world.

https://www.cnet.com/news/ikeas-rolling-out-a-brand-new-smart-home-lineup/

http://www.ikea.com/ms/sv_SE/customer-service/about-our-products/smart-lighting/index.html

"Trådfri" means 'wireless' in Swedish, and Ikea have so far announced this very aggresivly low-priced network-attached (Ethernet) "Ikea Trådfri Gateway" home automation hub in their "Tradfri" series, as well as a wireless Motion Sensor Kit (that have integrated light sensor too), a wireless Dimmer Remote (which is accelerator-based), a wireless multi-switch remote, and several smart light bulbs of different formats and even a few unique panel lights. All these products will then be released in most other contries worldwide too as Ikea steps up manufacturing (and irons our the initial software bugs I guess).

Ikea had already leaked news about this upcoming gateway/hub more than 6-months ago, during the summer or 2016, and at that time they also revealved that they will use ZigBee and keep validated access to the gateway/hub as open as possible, including providing an open API for this network-attached home automation hub.

Ikea in Sweden are first to post this news about the network-connected Home Automation Gateway / Hub, but again these products will become available worldwide. Here is link to the Swedish links:

http://www.ikea.com/se/sv/catalog/categories/departments/living_room/36812/ http://www.ikea.com/se/sv/catalog/products/40337806/ http://www.ikea.com/se/sv/catalog/products/80338960/ http://www.ikea.com/se/sv/catalog/products/80338941/ http://www.ikea.com/se/sv/catalog/products/80349888/

Link are in Swedish for Ikea Sweden site, but the PDF manuals on each page are available in English and many more languages, however they don't say much other than mounting instructions.

http://www.ikea.com/ms/en_US/img/buying_guides/fy17/april/Home_Smart_lighting_Buying_guide_APRIL1.pdf

Reason why I think that this news being interested is Ikea's aggresive pricing might them the first to make two-way communication home automation really affordable for almost everyone while still following all the electrical safety and wireless communications regulations in all countries, as they are today well known to have very low prices yet good manufacturing quality items.

UPDATE 1: Sound as Ikea Trådfri Gateway software uses the Cypress WICED IoT platform SDK (formerly Broadcom WICED IoT platform before acquired by Cypress http://www.cypress.com/internet-things-iot ) and have choosen to base their implementation on OMA (Open Mobile Alliance) and Eclipse recommended IoT protocol standards of those three logical components; CoAP (coaps) and DTLS layers of the LwM2M (Lightweight machine-to-machine) security model for IoT device management and protocol stack, using IPSO (IP for Smart Objects) Smart Object Guidelines provide a common design pattern. That is, looks as if the communication between Android/iOS app and the Gateway takes place via OMA Lightweight M2M (LwM2M) wrapped in CoAP with DTLS.

http://openmobilealliance.org/data-models-for-the-internet-of-things/

Update 2: Jaime Jiménez (from the company Ericsson) who is an active member of the IPSO Alliance’s working group and part of the team that published the IPSO Smart Object Guidelines, posted this great teardown of the Ikea Trådfri implementation:

http://jaimejim.github.io/tradfri/

For those who don’t know, LWM2M is a protocol built around CoAP and use for managing devices. So things like firmware upgrades, error reports, etc. Apart form the management interfaces, LWM2M also adds a very simple Object Model for managing those devices. IPSO expands that set of Objects so that you can have application information too (e.g. sensor readings, commands, etc). IPSO defines objects and resources that map to device properties.

Particular pay attention to the IPSO Light Control objects:

https://github.com/IPSO-Alliance/pub/blob/master/reg/xml/3311.xml

If you want to know more about the wealth of data models out there you can check the IoTSI Workshop as a reference.

https://www.iab.org/activities/workshops/iotsi/

LwM2M (Lightweight machine-to-machine) meanwhile is a system standard in the Open Mobile Alliance. It includes DTLS, CoAP, Block, Observe, SenML and Resource Directory and weaves them into a device-server interface along with an Objects structure based on IPSO Smart Object Guidelines.

https://connect2.io/open-mobile-alliance-lightweightm2m-oma-lwm2m/ https://www.eclipsecon.org/na2014/sites/default/files/slides/Eclipsecon%20NA14%20-%20One%20protocol%20to%20rule%20them%20all-%20(1).pdf http://openmobilealliance.org/constrained-application-protocol-coap-is-iots-modern-protocol/

IPSO provide common object model for interoperability of IoT Devices and Applications.

https://github.com/IPSO-Alliance/pub/blob/master/reg/README.md https://github.com/IPSO-Alliance/pub/tree/master/reg/xml https://github.com/IPSO-Alliance/pub/blob/master/reg/xml/3311.xml https://github.com/IPSO-Alliance/pub

[hvanderlaan]

The Ikea tradfri uses the iCoAP libs, this lib supports http/https proxy. Maybe in the future this will/could be enabled. When this is done the lib-coap / coap-client is not needed any more because you could talk to the http/https proxy.

https://github.com/stuffrabbit/iCoAP

If they enable this function the python module requests of urllib should be sufficent to talk with the hub.

Ik got also feedback from Ikea, they tolled me that from October the tradfri will have Apple's Home Kit support. So I thing they will open up the api :smile:

[Justblair]

https://www.hardill.me.uk/wordpress/2017/04/06/fist-pass-tradfri-mqtt-bridge/

A Java based mqtt bridge. Useful for a Openhab Add on writer?

Edit... Posted this in the wrong place - more than one window open

[kaikreuzer]

Did anyone actually try to reach out to IKEA and ask if they plan a public API documentation (similar to Philips hue)? It would make stuff much easier than having to guess how it works through reverse engineering...

[hardillb]

Yes: https://twitter.com/hardillb/status/848644863885271040

And half the fun is working it out for ourselves

[hvanderlaan]

I also had contact with Ikea, they say that in october they will add Apple's Home Kit Support. So there will be an api for this in the future. Also iCoAP can use a proxy but for now it is not enabled.

[Hedda]

I also used Ikea's contact-form a couple of days ago to ask if Ikea (Sweden) have any news to shere about their plans for an open/documentred API, but I have not recieved any reply to that as of yet.

Regardless I would advice everyone to write to Ikea and ask them to open up the Trådfri Gateway API.

http://www.ikea.com/ms/en_US/customer_service/contact_us/email_request.html http://www.ikea.com/gb/en/customer-service/contact-us/email/ http://www.ikea.com/ms/sv_SE/kundservice/kontakta_oss/maila_oss.html https://customersupport-ikea-nl-nl.custhelp.com/app/ask/ http://www.ikea.com/ms/en_HK/customer_service/contact_us/contact.html

Would be good if Ikea get more feedback so they understand that not everyone use Apple HomeKit.

[kaikreuzer]

Would be good if Ikea get more feedback so they understand that not everyone use Apple HomeKit.

I fully agree - I'll try to do my part to make IKEA aware of the importance of open APIs.

[vandenberghev]

I'm not sure it was in this thread, but I read that Ikea already confirmed they are going to release an open API...

Sources:

https://www.dailybits.be/item/ikea-tradfri-smart-lighting-app-netwerkhub/ (in dutch) "In vroegere interviews heeft IKEA beloofd om een open API interface te gaan aanbieden naar de Tradfri producten, zodat je deze kan connecteren met vele andere services. Momenteel is deze nog niet beschikbaar en werkt alles enkel lokaal. IKEA doet het stap voor stap met hun Tradfri ecosysteem."

http://www.domoticz.com/wiki/Ikea_Tradfri_Gateway "Ikea Trådfri Gateway comes with support for its own official Android and iOS apps from Ikea, however Ikea have also said that they are working on an open API for the Ikea Trådfri Gateway."

https://github.com/bwssytems/ha-bridge/issues/570 "Ikea had already leaked news about this upcoming gateway/hub more than 6-months ago, during the summer or 2016, and at that time they also revealved that they will use ZigBee and keep validated access to the gateway/hub as open as possible, including providing an open API for this network-attached home automation hub."

[andersmilton]

@hvanderlaan

I'm having the same problems as @sandyjmacdonald, I can read values through coap-client, but I can't seem to control my lights; I only get 4.00 as a response and neither the 5850 nor the 5851 values change.

I've tried controlling both the individual lights and the group, but I can't seem to get it to work. I've also tried the Python class with no success.

The commands I've tried for controlling the lights are: echo '{"3311":[{"5851":254}]}’ | coap-client -u "Client_identity" -k $KEY -m put "coaps://$IP:5684/15001/65537" -f - and echo '{"3311":[{"5850":1}]}’ | coap-client -u "Client_identity" -k $KEY -m put "coaps://$IP:5684/15001/65537" -f -

And for controlling the group: echo '{"5851":254}’ | coap-client -u "Client_identity" -k $KEY -m put "coaps://$IP:5684/15004/166682" -f - and echo '{"5850":1}’ | coap-client -u "Client_identity" -k $KEY -m put "coaps://$IP:5684/15004/166682" -f -

4.00 as a response for all of the above. I'm starting to think I've got a typo somewhere in there... But then again, the Python class doesn't seem to work for me either.

[sandyjmacdonald]

Try it the following way, @andersmilton:

coap-client -m put -u "Client_identity" -k "1a2b3c4d5e6f7g8h" -e '{ "3311": [{ "5850": 1 }] }' "coaps://192.168.0.10:5684/15001/65537"

[andersmilton]

OMG, works like a charm. Thanks @sandyjmacdonald!

[sandyjmacdonald]

:-) Took me a while to suss that too. Not sure why the echo and pipe isn't working?

[andersmilton]

Perhaps a character encoding problem or something? What platform are you on? I'm running it from a macOS terminal window through ssh to another Mac, so I guess there's plenty of room for translation errors...

[sandyjmacdonald]

I'm SSHing into a Pi Zero W from a macOS terminal window.

[andersmilton]

Looks like macOS could be the problem. Oh, well. As long as it works :)

[hvanderlaan]

This is correct, it's an issues with macOS, on a ubuntu box i can use the echo way but the payload argument of coap-client is a very nice solution to use

[vandenberghev]

My default OS X terminal encoding seems to be UTF-8. You can find out your encoding with the command locale. And I found this quite interesting:

"Now that we have checked the local terminal settings, we should do the same for hosts where we ssh into. Luckily, ssh can forward our locales settings, just append “SendEnv LANG LC_ALL” to ~/.ssh/config and check that your locale is also available on the remote host. Voila, you have a properly working terminal with defined locales." (found at https://benjamin-schweizer.de/unix-terminals-surviving-the-encoding-hell.html)

[cabo]

On Apr 12, 2017, at 12:45, Sandy Macdonald notifications@github.com wrote:

coap-client -m put -u "Client_identity" -k "1a2b3c4d5e6f7g8h" -e '{ "3311": [{ "5850": 1 }] }' "coaps://192.168.0.10:5684/15001/65537”

Actually, you can leave off that :5684, that is implicit in the coaps:

Grüße, Carsten

[sandyjmacdonald]

I've put together a tutorial about how to set all this up on a Pi, using @hvanderlaan's Python code, and showing how to use one of our (Pimoroni) Enviro pHAT boards as a light-responsive trigger, for those that might be interested.

https://learn.pimoroni.com/tutorial/sandyj/controlling-ikea-tradfri-lights-from-your-pi

[obgm]

You can now also omit the -u for coap-client: coap-client -m put -k "1a2b3c4d5e6f7g8h" -e '{ "3311": [{ "5850": 1 }] }' "coaps://192.168.0.10/15001/65537"

[andersmilton]

It doesn't seem to let me omit the -u, but I found an easy way of changing the color of the light by using the RGB hex values (f5faf6, f1e0b5 or efd275) like so:

coap-client -m put -u "Client_identity" -k "$KEY" -e '{ "3311": [{ "5706": "f5faf6" }] }' "coaps://$IP/15001/65537"

I don't think I've seen that posted here before. I apologise if I'm mistaken :)

[r41d]

I just implemented Color Changing (warm/normal/cold) for bulbs in the MQTT bridge from @hardillb: https://github.com/r41d/TRADFRI2MQTT/tree/features Awesome foundation by the way :+1: If my pull requests get accepted, this should be the most complete open source Tradfri tool yet =)

[kartben]

Some findings regarding /15011 a.k.a the gateway (mostly based on looking at the code of the Android app):

• a POST on /15011/9031 performs a soft reset
• a PUT on /15011/9034 is used for performing a firmware update, but it looks like this is not implemented yet (firmware payload is empty)
• a PUT on /15011/15012 can be used to perform many different operations (check OTA, commissioning, update time…) on the gateway, e.g update the current time by PUTting a timestamp with key 9060, etc.
• /15011/9033 is TBD
• /15011/9063 (POST) appears to be used for re-keying the gateway, as the code seems to involve the generation and transfer of a new PSK

[Justblair]

For those on linux based systems have a look at this...

https://www.hardill.me.uk/wordpress/2017/04/06/fist-pass-tradfri-mqtt-bridge/

It seems to rely on a coap library (hence the linux thing, unless I am mistaken) and creates a mqtt to tradfri bridge. Obviously it would be better if we had a binding that did it natively (and was cross platform)

[hardillb]

@Justblair it's all pure java so should run (nearly) anywhere

[matemaciek]

Some inside news from IKEA, nothing official, it can be changed, don't get attached to them:

• they're working on open API,
• new products are to appear in October,
• even more products in April next year.

I can't say, what products they're going to introduce, but let's just say, that I think that RGB bulb and controllable plug sockets would be a nice addition to the system (-:

[Justblair]

Oh I see. I misunderstood the coap library thing.

And yet again posted in the wrong place ...

[vidarlo]

On Thu, 13 Apr 2017, Maciej Sokołowski wrote:

I can't say, what products they're going to introduce, but let's just say, that I think that RGB bulb and controllable plug sockets would be a nice addition to the system (-:

Do you have any information regarding the number of devices? I've seen a maximum of ten devices connected - and this seems low in some cases.

[matemaciek]

There is a limit of ten groups per gateway and ten devices per group, so limitation is not that painful.

[vidarlo]

There is a limit of ten groups per gateway and ten devices per group, so limitation is not that painful.

Ok, so we can connect 100 devices theoretically?

Do you happen to know if the gateway supports observation, as per RFC7641? That would be a quite nice addition, as Ikea makes motion sensors and dimmers for Trådfri.

[hardillb]

Observations work, but observing a dimmer does not give the level changes, I've not worked out what it is yet, but I suspect it might be the battery level

[r41d]

RGB bulbs from IKEA would be awesome, there is also already a hint in the code: the COLOR_X and COLOR_Y constants in the Tradfri Android App. It's possible to express color with X and Y coordinates with the CIE 1931 color space: https://en.wikipedia.org/wiki/CIE_1931_color_space#CIE_xy_chromaticity_diagram_and_the_CIE_xyY_color_space The following values from the CoAP traffic were extracted:

public static final String COLOR_COLD = "f5faf6";
public static final String COLOR_COLD_X = "24930";
public static final String COLOR_COLD_Y = "24694";
public static final String COLOR_NORMAL = "f1e0b5";
public static final String COLOR_NORMAL_X = "30140";
public static final String COLOR_NORMAL_Y = "26909";
public static final String COLOR_WARM = "efd275";
public static final String COLOR_WARM_X = "33135";
public static final String COLOR_WARM_Y = "27211";

Changing color temperatures doesn't need any of the X or Y values, but if we look at this picture https://upload.wikimedia.org/wikipedia/commons/3/3b/CIE1931xy_blank.svg Not entirely sure how the scaling works, we have 0-1 in the linked picture but 5 digit integers in the CoAP traffic, 65535 is probably the maximum there. Cold = 24930,24694 -> 0.38,0.38. Both values are almost similar. This is near to white in the wikipedia picture. Normal = 30140,26909 -> 0.46,0.41. Greater values in both X and Y (though much more increase in X), it's probably somewhere in that yellow range. Warm = 33135,27211 -> 0.51,0.42. Again, much more increase in X than in Y, it's probably in the orange area. Does that sound reasonable?

[matemaciek]

Yeah @r41d , that seems like a good approximation.

I must find some time to try some colors in the middle of Tradfri bulbs range. Maybe we can set more than three colors via CoAP? I dissected one of my bulbs, it has two colors of emitters, warm and cold, and, of course, that makes three available color temperatures. But I'm wondering whether we can set them to different brightness to achieve the whole spectrum between cold and warm white.

[sandyjmacdonald]

You do indeed have full control over the colour temperature of the bulbs, and not just the three temps! Here's an example that ramps back and forth through the range:

import os
import time
import math

coap = '/usr/local/bin/coap-client'
securityid = 'xxxxxxxxxxxxxxxx'
hubip = '192.168.x.x'
lightbulbid = '65538'
tradfriHub = 'coaps://{}:5684/15001/{}'.format(hubip, lightbulbid)

while True:
    scale_factor = (1 + math.sin(time.time() * 2)) / 2
    payload = '{ "3311" : [{ "5709" : %s, "5710": %s }] }' % (25000 + int((8000 * scale_factor)), 25000 + int((2000 * (1 - scale_factor))))
    api = '{} -m put -u "Client_identity" -k "{}" -e \'{}\' "{}"' .format(coap, securityid, payload, tradfriHub)
    result = os.popen(api)
    time.sleep(0.01)

[balloob]

For the integration with Home Assistant we made the following Python lib: https://github.com/ggravlingen/pytradfri/ . The Python abstraction maps the API 1:1

These are the constants that we have been able to figure out thus far

[r41d]

@balloob You can also have a look at http://sprunge.us/CCQF (Also finally put them on gist: https://gist.github.com/r41d/5d62033f88b3046bccf406c9158d4e59) These are constants that I extracted from the Android Tradfri App, class com/ikea/tradfri/lighting/ipso/IPSOObjects.java @matemaciek also used them in his tradslator I think.

[sandyjmacdonald]

Hmm... everything seems to be broken for me. I wonder if IKEA pushed an update that somehow broke things. Not even querying the API with a get request from coap-client is working. :-(

[vidarlo]

Hmm... everything seems to be broken for me. I wonder if IKEA pushed an update that somehow broke things. Not even querying the API with a get request from coap-client is working. :-(

Which firmware version do you have? My GW has 1.1.0015 and works perfectly fine.

[Hedda]

FYI, Pimporoni posted step-by-step instructions for how pne could implement @hvanderlaan & @sandyjmacdonald Phyton code here:

https://learn.pimoroni.com/tutorial/sandyj/controlling-ikea-tradfri-lights-from-your-pi

They make it very easy to follow if someone who is not developer just want to try it out to test the functions.

[Hedda]

Ops, didn not mean to close this, reopening.

[sandyjmacdonald]

Have fixed my problems, @vidarlo. It seems to be that one of the bulbs got stuck while updating and that was causing no requests to get through. Had to reset everything, pair everything again, and it all works as expected now.

[possan]

All these things works great if you have lights and groups set up, i'm trying to read the value directly off a remote/dimmer, but i'm suspecting that it doesn't work like that...

Does the hub know if a remote is clicked on or does that communication happen directly to the lightsource then the lightsource broadcasts it's state to the hub so that the app can read it?

Or is the only way to do that to build a zigbee device?

[szokeptr]

@r41d I figured out that the colorX and colorY (5709 and 5710) range up to 10000, because in the CIE color space the values range between 0-1, and here they're probably just scaled up.

With this in mind, I created a homebridge plugin that has full support for the Ikea gateway and light bulbs, including dimming and accurate color changing, natively from the Home app: szokeptr/homebridge-tradfri-plugin

[r41d]

@szokeptr Hmm, did you look at my computations in my previous post and the correlation with the wikipedia picture? From that it looks much more likely that the maximum is 2^16-1=65535. But I can't say anything definitive about this as I didn't do any testing like you or @sandyjmacdonald did, should make up leeway. I think the most interesting part now, is which ranges are actually compatible to control the temperature.

[szokeptr]

@r41d yes, it looks like you're right, however this way my method is not that accurate. I am actually converting the hue and saturation values from the Home app to RGB then to CIE XY values. The function I am using is outputting values in the range of 0-1, and when I scaled that up to 0-100000 the lights are following the color changes almost dead accurate. I changed the scaling factor to 65535 and this way the colors are all in a much colder zone. I'll do some more testing on this.

[mariusobiegala]

@possan As described here: https://github.com/ggravlingen/pytradfri/issues/2: You can indeed listen (subscribe) to changes on a bulb with -s coap-client -u "Client_identity" -k {KEY} -s "60" "coaps://{IP}:5684/15001/65537" This will leave the coap-client running and pop out new messages if there are changes, e.g. by clicking on any button on the connected remote or control the bulb within the app.

[Hedda]

Jaime Jiménez ( @jaimejim from the company Ericsson) who is apparently an active member of the IPSO Alliance’s working group, and especially worked in the team that published the IPSO Smart Object Guidelines, posted this great teardown of the Ikea Trådfri implementation:

http://jaimejim.github.io/tradfri/

He also shared a couple of links to related articles he posted earlier:

http://jaimejim.github.io/lwm2m-yang/

http://jaimejim.github.io/coap-functionality-lwm2m/

Also cheackout the IoTSI Workshop as a reference:

https://www.iab.org/activities/workshops/iotsi/

Interestingly enough Michael Koster ( @mjkoster who is the main author of these IPSO Objects) posted a comment to Jaime's post staying: "Great move on IKEA's part here. Too bad we hadn't published the IPSO lighting objects yet. Maybe we can talk them into registering the new identifiers with OMNA.".

[Hedda]

@mjkoster is by the way the original author of iottoolkit - an IoT Toolkit with SmartObject API reference implementation and framework written in Python here which code could maybe be reused for this:

https://github.com/connectIOT/iottoolkit/

[obgm]

Hedda notifications@github.com writes:

FYI, Pimporoni posted step-by-step instructions for how pne could implement @hvanderlaan & @sandyjmacdonald Phyton code here:

https://learn.pimoroni.com/tutorial/sandyj/controlling-ikea-tradfri-lights-from-your-pi

Cool! One question: You are always using the -u "Client_identity". Is this required for Trådfri (since commit dc91eb9 from Apr 12, you can invoke coap-client without -u but I have no hardware to test against...)

[gobo-ws]

Hi all, I hope it is ok that I ask the following question here. It is a little bit off topic since it has nothing to do with the ha-bridge but I think this is a great thread for questions regarding IKEA's Trådfri API.

I have made a small shell script but it does not work. The echo output looks good and if I copy and paste the echo output manually in a terminal window it will work. I suspect that the error is caused by some wrongly escaped quote that is not shown with echo but I am new to shell scripts so I need some advice. Thanks in advance!

tradfri.sh

#!/bin/bash
set -x #echo on
exec coap-client -m put -u "\"Client_identity"\" -k "\"$1"\" -e ''\'{ "\"3311"\": [{ "\"5850"\": $2 }] }''\' "\"coaps://$3:5684/15001/$4"\"
exit

echo output & response from the coap-client after running the shell script:

$ sh tradfri.sh KEY 1 192.168.0.1 65537
+ exec coap-client -m put -u "Client_identity" -k "KEY" -e '{ "3311": [{ "5850": 1 }] }' "coaps://192.168.0.1:5684/15001/65537"
v:1 t:CON c:PUT i:dfb1 {} [ ]