search

dresden-elektronik / deconz-rest-plugin

deCONZ REST-API plugin to control ZigBee devices
BSD 3-Clause "New" or "Revised" License
1.88k stars 483 forks source link

New Xiaomi sensors #57

Closed NisseDILLIGAF closed 6 years ago

NisseDILLIGAF commented 6 years ago

So I bought some new Xiaomi sensors... one Xiaomi Aqara Temperature Humidity Pressure Sensor (which I got working, see below) and one Xiaomi Mi Smart Home Door / Window Sensors

There is a strange thing with the door sensor... it's got Manufacturer Code 0x0000 !? Maybe It's not an original?

anyways, this is what I got in deCONZ http://imgur.com/h6aSOtk http://imgur.com/IiAIAvw http://imgur.com/EhMNszo (gif of when I move the magnet from/to sensor)

I got the Xiaomi Aqara working after adding some code... this is the patch files for the Xiaomi Aqara Temperature Humidity Pressure Sensor https://gist.github.com/NisseDILLIGAF/fca6f06153a36bcd8f8b5cf80ab81605 I hope I got it right... It looks like it's working :) I get the three sensor values in REST

By the way... whats new in deconz-2.04.54-qt5.deb , it's not on github...

manup commented 6 years ago

There is a strange thing with the door sensor... it's got Manufacturer Code 0x0000 !? Maybe It's not an original?

Some products use manufacturer code 0x0000 which sometimes means they are white label products and where not branded by buying company.

The latest commits provide the VENDOR_NONE define which can be used in such cases.

I got the Xiaomi Aqara working after adding some code... this is the patch files for the Xiaomi Aqara Temperature Humidity Pressure Sensor https://gist.github.com/NisseDILLIGAF/fca6f06153a36bcd8f8b5cf80ab81605 I hope I got it right... It looks like it's working :) I get the three sensor values in REST

Cool thanks looks good, can you please open a pull request then I'll merge it into master :)

By the way... whats new in deconz-2.04.54-qt5.deb , it's not on github...

Commits are pushed now

NisseDILLIGAF commented 6 years ago

ok, so I tried adding { VENDOR_NONE, "lumi.sensor_magnet" }, to supportedDevices. Still nothing in REST... :( I guess I have to learn more how _ONOFF_CLUSTERID works... :)

donnib commented 6 years ago

@NisseDILLIGAF is this the one you have ? http://www.gearbest.com/smart-light-bulb/pp_257677.html or the new one http://www.gearbest.com/access-control/pp_626703.html, do you have other Xiaomi items other than the door sensor and the temperature sensor you mentioned here in this issue ?

NisseDILLIGAF commented 6 years ago

@donnib I have the old one, Original Xiaomi Smart Door and Windows Sensor , but I don't have it working in REST yet... The other Xiaomi sensors I have is the old temperature/Humidity sensor and the new with pressure... They are working in REST...

ebaauw commented 6 years ago

I ordered an Aqara door/window sensor from AliExpress. Their tracking claims it's arrived here by air, but I've yet to receive it. Sure hope we can get it to work. I'll post my experiences...

donnib commented 6 years ago

@NisseDILLIGAF thanks

@ebaauw that sounds terrific, let us know how it goes.

manup commented 6 years ago

Most (all?) of these sensors act as IAS_ZONE sensors, integration should be be possible with little adjustments.

I don't have the Xiaomi contact sensor yet but will order it soon. The temperature humidity sensor works very well till now (not sure what to do with the values though) :)

ebaauw commented 6 years ago

Received the sensor today. It's really cute, as small as it is.

I paired with deCONZ on the very first attempt. I don't think deCONZ reads it correctly; it only shows a 0x0000 basic cluster and a (server!) 0x0006 On/Off cluster. Reading the 0x0006 attributes fails. The virtual light on the node in the deCONZ GUI blinks blue when I align or remove the magnet, so it seems to do something.

Here's a screenshot of the Node Info: screen 1

And here's a screenshot of the 0x0000 Cluster Info: screen 2

manup commented 6 years ago

Can you please provide debug output by running deCONZ with logging enabled?

deCONZ --dbg-aps=1 --dbg-info=1

Would be interesting to see what commands are received when magnet is moved.

ebaauw commented 6 years ago

Wouldn't it need a client cluster to send commands? Could it be using attribute reporting instead of commands? I notice the 0x00 (OnOff) attribute of the 0x0006 cluster changes value immediately, when I move the magnet. It's false when there's contact and true when I take away the magnet. I only see only the following message in the output, each time I move the magnet:

16:57:38:121 APS-DATA.indication srcAddr: 0x00158d0001299e73, dstAddrMode: 2, profile: 0x0104, cluster: 0x0006, lqi: 255, rssi: -46

It doesn't seem to support reading the attributes: when I try reading them, I receive an error, and afterwards all attributes are greyed. When I double click the 0x00 attribute after restarting deCONZ, the attribute reporting window pops up, but reading the reporting config fails as well.

manup commented 6 years ago

Ah indeed, the other Xiaomi sensors also just use attribute reporting to update their state. In this case it's a bit weird, but ok I can put something together.

Question is, what should the device be?

I figure the ZHAOpenClose is more appropriate in general for door and window contact sensors!?

ebaauw commented 6 years ago

Yes, ZHAOpenClose.

I still wonder whether deCONZ reads the sensor correctly. There is hardly any info to be found on the Internet, but this French site lists a different device ID and a different set of clusters. Still, no battery cluster.

manup commented 6 years ago

For this specific device the important clusters are basic and on/off. In my experience only the Philips devices provide an usable battery cluster. Battery status for devices using CR**** batteries is usually quite inaccurate, the only useful information is if they report that battery needs to be changed. The sensors can detect voltage drops only shortly before battery dies.

ebaauw commented 6 years ago

Sniffing the ZigBee network, indeed, the Aqara sensor sends attribute reporting for attribute 0x0000 in cluster 0x0006.

When I move the magnet like crazy, I see an attribute reporting message for attribute 0xFF01 in the 0x0000 cluster. It's data type is 0x42 and the data contains elements 0 to 8.

abmantis commented 6 years ago

Not sure if it helps, but have you checked the implementation for SmartThings? They connect directly to the ST hub, and the guys over there have already reverse engineered it: https://github.com/a4refillpad/Xiaomi/tree/master/devicetypes/a4refillpad

ebaauw commented 6 years ago

v2.04.60: deleted the sensor from the network in the deCONZ GUI. Opened the network in the web app. Reset the sensor. It appears in the GUI and a resource is created in the REST API. I even received an event for it. The sensor is looking good, except that state remains empty:

{
  "config": {
    "on": true,
    "reachable": true
  },
  "ep": 1,
  "etag": "319ca4990bba547e27df402e3170e3a0",
  "manufacturername": "LUMI",
  "modelid": "lumi.sensor_magnet.aq2",
  "name": "ZHAOpenClose 2",
  "state": {},
  "swversion": "3000-0001",
  "type": "ZHAOpenClose",
  "uniqueid": "00:15:8d:00:01:xx:xx:xx-01-0006"
}

Maybe I'm too impatient, but I noticed that the GUI only showed the Basic Cluster at first. The OnOff cluster only appeared when I was moving the magnet. Might it be a timing issue?

Deleted the resource, removed the sensor from the GUI and retried pairing while moving the magnet. No change. The GUI reacts to moving the magnet, by showing the corresponding value of the OnOff attribute in real time.

Attached the output of deCONZ --dbg-info=2:
deCONZ.log.gz. Please let me know if you want me to turn on additional logging?

ebaauw commented 6 years ago

Hm, tried some more times, rebooted the Pi and I now see:

{
  "config": {
    "on": true,
    "reachable": true
  },
  "ep": 1,
  "etag": "a6948674df600ab63d112246de622493",
  "manufacturername": "LUMI",
  "modelid": "lumi.sensor_magnet.aq2",
  "name": "ZHAOpenClose 2",
  "state": {
    "lastupdated": "1969-12-31T23:00:00",
    "open": false
  },
  "swversion": "3000-0001",
  "type": "ZHAOpenClose",
  "uniqueid": "00:15:8d:00:01:xx:xx:xx-01"
}

The REST API created a default state object, but no events nor changes when moving the magnet. Also note the uniqueid: this time without the cluster?!

manup commented 6 years ago

I think I found the problem, incoming on/off reports diedn't update the sensor. New version will be uploaded in a few minutes

manup commented 6 years ago

Updated version is up now, please download again (still version 2.04.60).

http://www.dresden-elektronik.de/rpi/deconz/beta/deconz-2.04.60-qt5.deb

ebaauw commented 6 years ago

I did what you asked too literally: I downloaded it, and didn't see any difference. Luckily that changed after running dpkg -i...

I now get a state object:

{
  "config": {
    "on": true,
    "reachable": true
  },
  "ep": 1,
  "etag": "f9977f428306b34f9cacd1cff70f526f",
  "manufacturername": "LUMI",
  "modelid": "lumi.sensor_magnet.aq2",
  "name": "ZHAOpenClose 2",
  "state": {
    "open": false
  },
  "swversion": "3000-0001",
  "type": "ZHAOpenClose",
  "uniqueid": "00:15:8d:00:01:xx:xx:xx-01-0006"
}

and, more importantly, real-time events:

Wed Aug 02 2017 14:49:08: {"e":"added","r":"sensors","sensor":{"config":{"on":true,"reachable":true},"ep":1,"etag":"bde8a0e9793d09f52a2bab4712083ee4","id":"2","manufacturername":"LUMI","modelid":"lumi.sensor_magnet.aq2","name":"ZHAOpenClose 2","state":{},"swversion":"3000-0001","type":"ZHAOpenClose","uniqueid":"00:15:8d:00:01:29:9e:73-01-0006"},"t":"event"}
Wed Aug 02 2017 14:49:20: /sensors/2/state: {"lastupdated":"1970-01-01T00:00:00","open":true}
Wed Aug 02 2017 14:49:22: /sensors/2/state: {"lastupdated":"1970-01-01T00:00:00","open":false}
Wed Aug 02 2017 14:50:40: /sensors/2/state: {"lastupdated":"1970-01-01T00:00:00","open":true}
Wed Aug 02 2017 14:50:41: /sensors/2/state: {"lastupdated":"1970-01-01T00:00:00","open":false}
Wed Aug 02 2017 14:50:43: /sensors/2/state: {"lastupdated":"1970-01-01T00:00:00","open":true}

Note that there's no value for state.lastupdated. After rebooting it's created and the uniqueid is mutilated:

{
  "config": {
    "on": true,
    "reachable": true
  },
  "ep": 1,
  "etag": "184869b1aef27711b99573f2f5fff61f",
  "manufacturername": "LUMI",
  "modelid": "lumi.sensor_magnet.aq2",
  "name": "ZHAOpenClose 2",
  "state": {
    "lastupdated": "1969-12-31T23:00:00",
    "open": true
  },
  "swversion": "3000-0001",
  "type": "ZHAOpenClose",
  "uniqueid": "00:15:8d:00:01:xx:xx:xx-01"
}

And the events show:

Wed Aug 02 2017 14:58:42: /sensors/2/state: {"lastupdated":"1969-12-31T23:00:00","open":true}
Wed Aug 02 2017 14:58:44: /sensors/2/state: {"lastupdated":"1969-12-31T23:00:00","open":false}
Wed Aug 02 2017 14:58:45: /sensors/2/state: {"lastupdated":"1969-12-31T23:00:00","open":true}
Wed Aug 02 2017 14:58:47: /sensors/2/state: {"lastupdated":"1969-12-31T23:00:00","open":false}
manup commented 6 years ago

Yes timestamp 14:14, if you observe ZCL attribute reporting from the sensor in the sniffer for the onoff cluster, the open state should be updated and events should be thrown.

manup commented 6 years ago

I'm not sure if the sensor creates the needed binding on its own, deCONZ doesn't create a binding yet.

ebaauw commented 6 years ago

I forgot to run dpkg - see my updated comment.

ebaauw commented 6 years ago

I'm not sure if the sensor creates the needed binding on its own, deCONZ doesn't create a binding yet.

I think it does - the OnOff attribute in the deCONZ GUI already reflects the magnet in real-time before v2.04.60. And, as mentioned in my corrected comment, I do get web socket notifications!

manup commented 6 years ago

Ah okay, good that events are now received, I'll fix the uniqueid and lastupdated attributes in the next version. Thanks again for testing :)

ebaauw commented 6 years ago

You're most welcome.

I tried some simple rules:

{
  "1": {
    "actions": [
      {
        "address": "/lights/1/state",
        "body": {
          "on": true
        },
        "method": "PUT"
      }
    ],
    "conditions": [
      {
        "address": "/sensors/2/state/open",
        "operator": "eq",
        "value": "true"
      }
    ],
    "created": "2017-08-02T13:16:42",
    "etag": "961efcc0123939d440e03e455f5d59ee",
    "lasttriggered": "2017-08-02T15:17:35",
    "name": "open",
    "owner": "philipshue",
    "periodic": 0,
    "status": "enabled",
    "timestriggered": 75
  },
  "2": {
    "actions": [
      {
        "address": "/lights/1/state",
        "body": {
          "on": false
        },
        "method": "PUT"
      }
    ],
    "conditions": [
      {
        "address": "/sensors/2/state/open",
        "operator": "eq",
        "value": "false"
      }
    ],
    "created": "2017-08-02T13:17:09",
    "etag": "67f4943040640c6d003383034b03dc11",
    "lasttriggered": "2017-08-02T15:18:33",
    "name": "close",
    "owner": "philipshue",
    "periodic": 0,
    "status": "enabled",
    "timestriggered": 400
  }
}

They work as expected. Of course, the rules still get triggered continuously (see timestriggered), so we really need lastupdated (or different rule triggering logic in line with the Philips Hue bridge).

NisseDILLIGAF commented 6 years ago

I cant get this to work with my 'original' door sensor that has Manufacturer Code 0x0000...? I tried just adding { VENDOR_NONE, "lumi.sensor_magnet" },, but it's not added to REST API...

manup commented 6 years ago

From the screenshots it looks that the ZigBee Node Descriptor was not received, hence the 0x0000. Can you please retry to add the sensor:

NisseDILLIGAF commented 6 years ago

I cant delete it via REST because it's not there.. I've deleted it in deCONZ, removed battery and tried adding it several times... no luck.

manup commented 6 years ago

Hm if the node descriptor is not read, the plugin will not create the API sensor.

Here are more things to check:

NisseDILLIGAF commented 6 years ago

I'll try that tomorrow, it's getting late here, going up early for work..

manup commented 6 years ago

Just the lights or all ZigBee devices?

All devices which show as yellow node in deCONZ -- mostly this are the lights.

NisseDILLIGAF commented 6 years ago

Ok, i turnd off all lights, but no luck... still the same when I connect..

runningman84 commented 6 years ago

I am curious to know if you share his opinion: https://community.hom.ee/t/xiaomi-mi-smart-home-serie/1028/18

manup commented 6 years ago

I am curious to know if you share his opinion: https://community.hom.ee/t/xiaomi-mi-smart-home-serie/1028/18

I totally agree that the Xiaomi sensors are very strange and anything but standard.

Therefore I do understand their decision to not support them, same is true for Philips with IKEA, it burns a lot of time to integrate things which should work but don't play by the rules.

For the various reasons he mentioned deCONZ has included some quirks to learn more about devices in non standard ways. Since the sensors don't support OTA updates they will be around for some time and should be supported by Xiaomi gateways, hence it's okay to support them in deCONZ.

In short if devices can be integrated and there is reasonable demand, then we very likely will integrate them.

ebaauw commented 6 years ago

I am curious to know if you share his opinion: https://community.hom.ee/t/xiaomi-mi-smart-home-serie/1028/18

He's right, technically, and no joy whatsoever. As I prefer being happy over being right, I wholeheartedly applaud and support (where I can) deCONZ approach.

I'm no ZigBee specialist. I'm not affiliated with any vendor. I'm just a hobbyist playing with home automation. In my experience there's no such thing as "a ZigBee standard". Even between officially ZLL certified lights, there's noticeable differences:

So, as a gateway/bridge, you have to make a fundamental choice:

  1. Strictly adhere to the standard, supporting only the functionality that can be discovered;
  2. Fully support known (whitelisted) devices, including the functionality that cannot be discovered.

In my experience, each vendor (Philips, OSRAM, innr, IKEA) takes the second approach, but only for their own devices. They take the first approach for devices by other vendors, and seem to prefer finger pointing over providing end-user functionality.

In short if devices can be integrated and there is reasonable demand, then we very likely will integrate them.

I'm very happy to see that deCONZ (or at least @manup) is taking the second approach, also for devices by other vendors (see also the last paragraph in https://github.com/dresden-elektronik/deconz-rest-plugin/issues/20#issuecomment-289249805). I think that's the Open Source spirit.

It would seem like Homee decided on the first approach.

NisseDILLIGAF commented 6 years ago

Manufacturer Code is still 0x0000 and it wont get in to rest api..

if I try to read basic cluster, RaspBee will lock up, the green light on the board will stay on and I can't do anything. I have to shutdown and pull the power...

Here is the full log ( I hope there isn't any sensitive info in there ) https://pastebin.com/7ZXmDzLT I've highligted all lines with device 0x00158D0001837326

manup commented 6 years ago

Cool thanks I'll check the logs to see whats happening here.

Does the sensor node still blink blue when you move the magnet (meaning deCONZ is receiving commands)?

If yes I can adjust the setup so that the sensor can be created from what is known (basic cluster modelid, mac address) without needing to read node descriptor.

NisseDILLIGAF commented 6 years ago

No, after i get it into deCONZ the blue light stops.. but it changes attributes true/false in cluster 0006 when i move the magnet

Thanx for great support!!!

NisseDILLIGAF commented 6 years ago

Alright!!! My Xiaomi door sensor works after commit https://github.com/dresden-elektronik/deconz-rest-plugin/commit/5c266a63809c6d2383774934b933661ec2464af9 Great work... and THANX again for great support @manup

should I close this issue?

manup commented 6 years ago

Cool :) hopefully this will solve similar problems with other devices as well. For now it can be closed.

I've ordered some more Xiaomi sensors to check what can be improved, the PIR is still missing.

NisseDILLIGAF commented 6 years ago

Hmmm ... having some difficulties with my IKEA lights/motion sensors... noting is working... I restarted my Pi and the lights are working... I had to pair one of the motion sensors again...

I think everything is working as it should now... strange? I'll close this issue... Thanx again! This is the best support I ever had on a product!!! Great work!

snozzlebert commented 6 years ago

Thanks @manup, I added my original Xiaomi Door Sensor (lumi.sensor_magnet) today without any problems.

ebaauw commented 6 years ago

Just added support for the Aqara door sensor to homebridge-hue. I can now use it in HomeKit!

I'd like to add support for the other Xiaomi sensors as well. @snozzlebert, @NisseDILLIGAF, could you please post what these look like in the REST API? What's the output of a GET of the /sensors resource? I suppose the temperature/humidity/pressure sensor has three associated resources in /sensors (like the Hue motion sensor)? Thanks!

snozzlebert commented 6 years ago

Temperatue (Original Xiaomi):

{
    "config": {
        "on": true,
        "reachable": true
    },
    "ep": 1,
    "etag": "baceb383ac8e25019e439fea82ed501e",
    "manufacturername": "LUMI",
    "modelid": "lumi.sensor_ht",
    "name": "Temperature",
    "state": {
        "lastupdated": "2017-08-05T11:03:18",
        "temperature": 2411
    },
    "swversion": "3000-0001",
    "type": "ZHATemperature",
    "uniqueid": "00:15:8d:00:##:##:##:##-01-0402"
}

Humidity (Original Xiaomi, part of the temperature sensor, so it has the same prefix in uniqueid as the temperature sensor):

{
    "config": {
        "on": true,
        "reachable": true
    },
    "ep": 1,
    "etag": "baceb383ac8e25019e439fea82ed501e",
    "manufacturername": "LUMI",
    "modelid": "lumi.sensor_ht",
    "name": "Humidity",
    "state": {
        "humidity": 5624,
        "lastupdated": "2017-08-05T11:03:18"
    },
    "swversion": "3000-0001",
    "type": "ZHAHumidity",
    "uniqueid": "00:15:8d:00:##:##:##:##-01-0405"
}

Original Xiaomi Door/Window sensor (manufacturername and swversion are missing, maybe a bug?)

{
    "config": {
        "on": true,
        "reachable": true
    },
    "ep": 1,
    "etag": "dd0b4d6a845420b02354e398acd57ce2",
    "modelid": "lumi.sensor_magnet",
    "name": "Door",
    "state": {
        "lastupdated": "2017-08-05T11:14:06",
        "open": true
    },
    "type": "ZHAOpenClose",
    "uniqueid": "00:15:8d:00:##:##:##:##-01-0006"
}
ebaauw commented 6 years ago

Thanks, @snozzlebert. Is this the temperature/humidity sensor, without pressure?

snozzlebert commented 6 years ago

Correct, the 'old' sensor had no pressure sensor. After buying these sensors I found out that Aqara was the new line of these sensors. What is your experience with the Aqara temperature sensor? My 'original' one doesn't update very often. Maybe once in 8 hours. Does it only update on significant changes?

ebaauw commented 6 years ago

What is your experience with the Aqara temperature sensor?

I only have the Aqara door/window sensor. I already get temperature through the Hue motion sensor, not sure if I'm ready to invest in more sensors, just for the added humidity (and air pressure). I might get one for the bathroom, though.

My 'original' one doesn't update very often. Maybe once in 8 hours. Does it only update on significant changes?

Normally, this would be configured through the ZigBee attribute reporting configuration. In the deCONZ GUI, select the Temperature (0x0402) cluster. Then in the Cluster Info panel, double click on the Measured Value (0) attribute. Enter the desired values and press write config. The Hue motion sensor uses MinInterval: 10, MaxInterval: 300, Change: 20, meaning: report once every 5 minutes, and when the temperature changes by more than 0.2 degrees, but at most once every 10 seconds.

ebaauw commented 6 years ago

I'll fix the uniqueid and lastupdated attributes in the next version.

Indeed, they're fixed in v2.04.61. Thanks, @manup!

ebaauw commented 6 years ago

New toys from China just arrived! Some more Xiaomi Aqara door/window sensors, a Xiaomi Aqara smart wireless switch, and two Xiaomi Aqara weather sensors. The last two use the same CR2032 batteries as the IKEA remote and motion sensor (which you can get ridiculously cheaply at IKEA). The door/window sensors use a smaller CR1632.

For pairing it's best first to reset the sensor (hold the reset button until the led blinks blue) and then quickly open the network. When you briefly press the reset button, the sensors report attributes, not just for the value, but also the model and manufacturer in the basic cluster. deCONZ creates the nodes without any issues, and the REST API plugin creates the resources after the relevant data has been read (might need to interaction with the sensor's button, to keep it awake).

The door/window sensor sends attribute reporting for open and close events.

Unlike the original Xiaomi button (see #89), the Aqara smart wireless switch only reports when releasing the button (not when pressing it), so deCONZ issues only 1002 buttonevents. Also the 0x0006/0x0000 attribute in the deCONZ GUI doesn't change and always shows true.

As expected, the REST API plugin creates three resources for the weather sensors: ZHATemperature, ZHAHumidity, and ZHAPRessure, corresponding to the 0x0402, 0x0405, and 0x0403 clusters. As expected, neither cluster supports Read attributes, but all three send Report attributes commands. For 0x0402, it's attribute Measured value (0x0000), an int16, with the measured temperature in 0.01 degrees Celcius. For 0x0402, it's attribute Measured value (0x0000), an uint16, with the measured humidity in 0.01 %. For 0x0405, it's three attributes: Measured value (0x0000), an int16 with the measured pressure in hPa (or mbar); Scale (0x0014), an int8, with value 255 (0xff - I think that's -1?); and Scaled value (0x0010), and int16 which is 10x the Measured value, or rather Measured value seems to be trunc(Scaled value * 10^(Scale)). Not sure how accurate these sensors are, so I wouldn't bother with anything but Measured value.

I don't get web socket notification for the ZHAPressure, and the state object in the sensor resource remains empty. There's no Added ZCLvalue 0x0403/0x000 message in the log. After restarting deCONZ, state.pressure is populated, but remains 0. state.lastupdated is 1969-12-31T23:00:00.

deCONZ reads the ZigBee attribute as uint16 instead of int16, https://github.com/dresden-elektronik/deconz-rest-plugin/blob/master/de_web_plugin.cpp#L3325. @manup, could that be the cause?
Related: I see state.temperature, state.humidity, and state.pressure are all defined defined as int32 in deCONZ (https://github.com/dresden-elektronik/deconz-rest-plugin/blob/master/resource.cpp#L96), as are state.buttonevent and state.status. I don't suppose that would be an issue though, as temperature and humidity work.