Enhancement: add history to sensors

[ebaauw]

Thanks for the hint, @krocko!

See: https://github.com/simont77/fakegato-history

[honkmaster]

This enhancement would be very welcome. Thanks for your work.

[oynek]

Highly appriciated!

[ebaauw]

v0.5.68 adds history to OpenClose sensors. I only tested it with my Xiaomi Aqara Door sensors, but it should work for CLIPOpenClose as well.

I think I should be able to add something similar for the Presence sensors. The weather / temperature sensors will be more challenging due to homebridge-hue's current design. The power plugs obviously need power measurement support in deCONZ first.

[iantait]

This is good stuff! (Xiaomi Aqara door sensors)

[ebaauw]

Check out v0.5.69. It should add history to the Xiaomi Aqara weather sensor and to all Motion sensors.
Still a few glitches, though:

• Eve also shows an empty history for the Temperature for the Hue motion sensor. Not sure if I can fix that, without exposing the motion sensor as two accessories (and once again running into the max 99 accessories per HomeKit bridge limit);
• History for MultiCLIP weather sensors containing Temperature, Humidity, Pressure, but also other services only work, if the weather services are discovered first, i.e. if they have the lowest sensor resource IDs.
• Eve also shows an empty history for the Power of a MultiCLIP weather sensor.

[to0b]

Thanks! Is it possible to add the history also for the old round Xiaomi Temp/Hum Sensor?

[iantait]

I upgraded to v0.5.69 and unfortunately (for debugging) upgraded the Eve app (3.1/943) around the same time, and now the history info isn't appearing or appearing incorrectly in Eve - not showing all the events.

[ebaauw]

I'm seeing the same, see https://github.com/simont77/fakegato-history/issues/33. Not sure if it's related to Eve 3.1 or if I'm doing something wrong.

[oynek]

Just upgraded to latest Eve/homebridge-hue. I have several Hue motions sensors installed but I can't see any history for these sensors in Eve app.

Where are these statistics supposed to be? If I long-tap on one of those motion sensors in Eve app all Hue motions sensor results (temperatue, lux, motion) show up - without any history.

Furthermore Homebridge does not seem to make any entries. The logs do not show any entries regarding like all your installs do.

Am I missing a point?

[ebaauw]

I think I found (most of) the issues, see https://github.com/simont77/fakegato-history/issues/33#issuecomment-362798453.

Currently executing a long-running test. I also need some more changes to cope with other characteristics for Duration and Sensitivity (which take different values than the Hue motion sensor). Hope to release new version this weekend.

Thanks! Is it possible to add the history also for the old round Xiaomi Temp/Hum Sensor?

I'll try and add it. Maybe I need to expose a fake Air Pressure Sensor service for the sensor.

History for MultiCLIP weather sensors containing Temperature, Humidity, Pressure, but also other services only work, if the weather services are discovered first, i.e. if they have the lowest sensor resource IDs. Eve also shows an empty history for the Power of a MultiCLIP weather sensor.

I won't be fixing these. I will add something to ph.sh or homebridge-hue-utils to create a multi-CLIP weather sensor (and temperature/humidity sensor for the Hue bridge).

[ebaauw]

Could you guys try v0.5.71? Motion and Door sensors should work.

Note that the history for the Temperature Sensor of the Hue motion sensor is empty.

Weather sensors should work as well, provided the ZHAPressure or CLIPPressure resource is the first or second resource on the gateway. If not, values of the first resource will be all 0. By default, the order of the resources for the Xiaomi Aqara weather sensor is temperature, humidity, pressure, so the temperature will be all zeroes. You can change the sensor resource ID only by patching the deCONZ REST API plugin database.
I intend to fix this, also to support temperature/humidity sensors, but this requires some serious refactoring.

[ebaauw]

Just released v0.6.0, which bases weather history on Humidity instead of on Pressure. This version should support:

• Full history for the Xiaomi Aqara Weather sensor (with default order of resources: Temperature, Humidity, Pressure);
• History for the Xiaomi Mi Temperature/Humidity sensor (untested - @to0b could you please test if this works);
• History for multi-CLIP temperature/humidity/pressure or temperature/humidity sensors.

Note that I had to change the Serial Number (and UUID derived from that) for multi-CLIP sensors, to support history. Consequently, after upgrading, HomeKit will see existing multi-CLIP sensors as new accessories. You need to re-create any HomeKit room assignments, groups, scenes, and automations that contained a multi-CLIP sensor.

[CooperCGN]

Is that impacting my Hue motion sensors?

[ebaauw]

No, only multi-CLIP sensors (multiple CLIP sensor resources that are exposed as a single accessory).

[iantait]

0.6.0 has been running since yesterday evening - all looks good on the door sensors - well done.

[ebaauw]

Thanks to the folks at fakagato-history, v0.6.2 preserves history when homebridge restarts.

[vbyjsue]

Extremely useful updates that makes my node-red dashboard almost obsolete. All thats missing is the power sensor. As I understand it needs to be implemented in deconz first? There is no way I can post my powerreadings to a custom-sensor and get it showing in eve? Thanks for an amazing work.

[ebaauw]

There is no way I can post my powerreadings to a custom-sensor and get it showing in eve?

Interesting idea. I'm still thinking about how best to expose the power readings in deCONZ: as an additional state.power attribute to the /lights resource, or as a separate ZHAPower /sensors resource. In the second case, we could add a CLIPPower as well.

Today the electricity network installed a smart power meter in my home, which, as I understand, reports the current power consumption every 10 seconds once I have the right cable to connect it to my Raspberry. I might eventually create a separate homebridge plugin for it, but putting the values read to a CLIP sensor would be a quick win (cf. the pre-homebridge-ws virtual weather station).

[ebaauw]

Added history for power consumption to v0.6.5. Works brilliantly for the Heiman plug; the OSRAM plugs report nothing useful, unfortunately. Note that, currently, deCONZ reports consumption through state.power in the /lights resource, so no CLIP version yet.

[vbyjsue]

Thats great news. I'm not sure what to do with my readings though. I read my powerconsumption by an esp8266 with a photoresistor that counts the "blinks" on the central unit. Tried PUT lights/{id}/state {"power":1000} but I realize that's not the right way.

[ebaauw]

No, you would need a CLIPPower sensor, which homebridge-hue would expose as Switch or Outlet to HomeKit (probably with a non-functional or read-only On characteristic), incl. the fakegato history. You would then be able to PUT the power to this sensor state.

I’ll see what I can come up with next weekend. I’m curious to see if this setup can handle a PUT every 10 seconds. Currently, I keep track of the time of the last PUT and update the total consumption on each PUT (elapsed time times previous value).

[to0b]

Hey Erik,

with 0.7.0 the eve app don't get the values from the new storage files. I only see new entries when the app is opened. The plugin writes the correct values to the files.

homebridge-hue.json.gz

homebridge.log.gz

I‘ve downgraded to 0.6.5 and everything works well.

[ebaauw]

@to0b, I'm seeing the same, and @Krocko has also reported this on Slack. I'm not sure what's causing this, let alone how to fix it. I'm investigating, but the Elgato history remains kind of a mystery to me...

For now, I've set homebridge-hue@latest back to v0.6.5 and added a warning to the release notes.

[iantait]

I was seeing the same, but after removing homebridge-ws, and the persist json files and waiting a bit, it automagically started working again. Is it anything to do with naming the files with the mac address?

From: Erik Baauw [mailto:notifications@github.com] Sent: 26 February 2018 16:58 To: ebaauw/homebridge-hue Cc: Ian Tait; Comment Subject: Re: [ebaauw/homebridge-hue] Enhancement: add history to sensors (#277)

@to0bhttps://github.com/to0b, I'm seeing the same, and @Krockohttps://github.com/krocko has also reported this on Slack. I'm not sure what's causing this, let alone how to fix it. I'm investigating, but the Elgato history remains kind of a mystery to me...

For now, I've set homebridge-hue@latest back to v0.6.5 and added a warning to the release notes.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/ebaauw/homebridge-hue/issues/277#issuecomment-368571066, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJOi1XPCCIFR64BHcQSHDM_ssQ3K65ITks5tYuKdgaJpZM4RxRI1.

[iantait]

Ah - maybe I had the app open too...

[ebaauw]

It's not related to the changed filenames, but to the use of the fakegato-timer. On my test environment (connected to the production deCONZ) I did the following:

• shutdown homebridge;
• remove all history files;
• remove the Eve app (make sure there's no history left in the app);
• restart homebridge;
• re-install Eve.

On homebridge-hue@0.6.5 history works, on homebridge-hue@0.7.0, it doesn't. On the v0.7.0 code, with only the fakegato-timer enabled it works again. I'll publish 0.7.1 shortly to install it on my production environment, to see how it runs overnight.

[ebaauw]

Ok running 0.7.1 on production now. I did not delete the history files this time. I uninstalled/reinstalled Eve on my work iPhone (same iCloud account) and history seems to work right away. On my iPad, I see the new entries but with the wrong time. I my private iPhone, I don't see the new entries. I deleted the history from Eve on my iPad earlier this evening, on my private iPhone Eve has extensive history (of over a week).

When restarting Eve after some 10 minutes (when the fakegato-timer has "flushed" the history entries), my private iPhone now shows new entries, at the correct time, but it has re-timed the historic entries.

This strengthens my suspicion that the issue is related to the time synchronisation being off between Eve and the fake Elgato device. The real Elgato devices sync their clocks with the Eve app. The app won't show (what it thinks are) future-dated entries. Instead of uninstalling/reinstalling Eve, you can select Edit|Delete all events and restart Eve. That seems to force Eve and the accessory to re-sync, and now the lost entries magically appear - at the correct time. Note that you need to do this for each accessory.

Apparently, the fakegato-timer has some side effect I don't yet understand. Maybe it's just that I write the first entry too soon, before the time sync entry is available. I need to analyse further, but not today.

As I said, I'll run 0.7.1 overnight and see if it holds. Until I'm satisfied, I'll keep 0.6.5 as latest version. If you want to help testing 0.7.1, you can install it through sudo npm -g i homebridge-hue@0.7.1.

[to0b]

I‘m running 0.7.1 since two hours. Previously I‘ve deleted all history entries in the eve app. Now it seems that everything works fine without gaps.

I have another question about the history: Is the history characteristic implemented for the hue motion temperature sensor? I‘ve noticed that the history is working for the motion characteristic but the temperature history is empty. I have in mind that there was a problem with mixing the eve characteristics for weather/room and motion?

[ebaauw]

Is the history characteristic implemented for the hue motion temperature sensor?

It's not a characteristic; it's a separate service with a bunch of characteristics. There's some real magic at work for Eve to match the value of some of these history characteristics to the regular characteristics in other services of the same accessory. They have read the values from real Elgato devices, but they haven't reversed engineered the magic. As there's no Elgato device that combines motion and temperature, we don't know if Eve would even support this, let alone what the corresponding magic should look like.

I still need to test if maybe Eve would work with two history services in a single accessory, but I don't hold high hopes.

Alternatively, I could expose the Hue motion sensor as two accessories, one for the Motion and Ambient Light sensors and one for the Temperature sensor, both with the corresponding history service. However, I cannot do that easily in the current design of homebridge-hue (which bundles all services of a single device into a single accessory, uniquely identified by the mac address). Also, as I have 11 Hue motion sensors, this would cause me to hit the 99 accessory limit, so I would need to setup two separate homebridge instances. For now, I'm busy adding smart plug support in deCONZ and then exposing these in homebridge-hue with history, like Eve Energy devices.

[ebaauw]

I think the only thing still open is providing history for both the Motion Sensor and the Temperature Sensor for the Hue motion sensor.

Currently, history is supported for only one sensor at a time. By default this is for the Motion Sensor, but if you add "excludeSensorTypes": ["ZLLPresence"] to config.json (or blacklist the ZLLPresence/ZHAPresence /sensors resource), the Temperature Sensor history is created.

If you run multiple homebridge instances (e.g. because you're hitting the 99 bridged accessories limit), you could add "excludeSensorTypes": ["ZLLPresence", "ZLLLightLevel"] to the first config.json and "excludeSensorTypes": ["ZLLTemperature", "ZHAHumidity", "ZHAPressure"] to the second.

While technically possible, I think the two sensors should not be combined in a single history log: the motion sensor needs entries each time the status changes where the temperature sensor only needs periodic entries. For now this is academic, because we don't know how to tell Eve that both are in the log anyway.
I still need to check whether Eve will handle correctly multiple history logs for one accessory, but I don't hold high hopes.
I might need to expose the Hue motion sensor as two accessories to get history for both Motion and Temperature in one homebridge instance.

[GatoPharaoh]

Is it possible to stop the history logs to homebridges output? at the moment, every entry added will be logged which makes my log pretty confusing.

thanks in advance

[ebaauw]

homebridge-hue issues a log message for every Interaction with HomeKit, I don’t see how that’s confusing. As per homebridge setup, log and debug messages are written to stdout, warning and error messages are written to stderr. You might want to capture these in separate log files.

[ebaauw]

v0.8.4 adds temperature history for the Hue motion sensor, see https://github.com/ebaauw/homebridge-hue/releases.

[iantait]

Awesome. 0.8.4 loaded, will check it out. Thanks!

[GatoPharaoh]

I'm specifically talking about the fakegato history logs. Before the introduction of the history function, the log would contain status changes or faults etc. With the introduction of fakegato, every singly entry is logged. Which means that at least every ten minutes all my devices will log the fakegato entry. By confusing, I mean I'm losing oversight due to those history logs.

[ebaauw]

@GatoPharaoh, I'll see if I can change the history entry messages to debug messages in the next version.

[GatoPharaoh]

awesome. thanks for your work!

[ebaauw]

I'll see if I can change the history entry messages to debug messages in the next version.

v0.8.5.

[ooii]

Hi,

I'm running the latest version of homebridge-hue with a decent gateway and, as you can see, cannot get temperature history with the hue sensors in Eve app. Is there something specific to do to have data history? Thanks.

[ebaauw]

Did you set hueMotionTemperatureHistory? See https://github.com/ebaauw/homebridge-hue/releases/tag/v0.8.4.

[ooii]

No 😞 Did not see it in the wiki/conf/keys. Thanks for the help.

[ebaauw]

Good catch, I added it to the Wiki, https://github.com/ebaauw/homebridge-hue/wiki/Configuration.

[ooii]

It's working now. I have temperature history since the moment I added this parameter to the conf file (today) but I have motion history since several days.

[tomky]

I setup a CLIPTemperature sensor in deCONZ but no history is displayed in Eve. What am I missing? I also set the hueMotionTemperatureHistory to true but no effect.

[jaydisc]

I, too, can't get Eve to present a history.

I've paired it down to this to no avail:

    {
        "platform": "Hue",
        "users": {
            "001788xxxxxxxxxx": "xxxxxxxxxxx"
        },
        "sensors": true,
        "excludeSensorTypes": [],
        "hueMotionTemperatureHistory": true
    }

Is anything else required? I'm using a Hue Bridge with Hue Motion Sensor.

UPDATE: I've gotten it to work by adding "nativeHomeKitSensors": false.

[ebaauw]

@tomky, @jaydisc do you see the sensor at all?

If no, double-check config.json: do you expose the CLIP sensor? If you have a v2 (square) Hue bridge, you need to set nativeHomeKitSensors to false to expose the Hue motion sensor.

If yes, could you please attach the debug dump file (see https://github.com/ebaauw/homebridge-hue#debug-dump-file) and a screenshot of the sensor in Eve?

[jaydisc]

I can get the history view for the motion sensor, but not the temperature.

homebridge-hue.json.gz

I followed the instructions in https://github.com/ebaauw/homebridge-hue/issues/277#issuecomment-370188300 to no avail ("excludeSensorTypes": ["ZLLPresence"]).

[ebaauw]

I assume the Motion Sensor temperature sensor is exposed by the Hue bridge itself? Can you double-check in the Accessories view (under Settings) in Eve?

I followed the instructions in #277 (comment) to no avail ("excludeSensorTypes": ["ZLLPresence"]).

That workaround should cause the Motion sensor resource no longer to be exposed, so homebridge-hue will create history for the temperature sensor instead. It became obsolete with the introduction of hueMotionTemperatureHistory. I assume your screenshot is from running without this setting?

Nothing weird in your bridge configuration and config.json looks fine. Did homebridge-hue create the history file for the temperature sensor? Should be history_00178801032907C8-T.json in ~/.homebridge/accessories.

Not sure what's going on here. Since you're only exposing a single motion sensor, could you please:

• Remove homebridge from HomeKit (in Eve in the Accessories view);
• Stop homebridge;
• rm -r ~/.homebridge/persist ~/.homebridge/accessories;
• Start homebridge -D and capture the log (see https://github.com/ebaauw/homebridge-hue#debug-log-file);
• Re-add homebridge to HomeKit.

[jaydisc]

Good to know about ZLLPresence.

After performing those steps, I can happily get temperature history from the Hue Motion Sensor now. Thank you very much.

[thecem]

Hi is there a possibility to have for the Philips Hue Motion Sensor the temperature history in eve-app? Every temp sensor shows the history but not the hue motion sensors (hombridge-hue: 0.11.8 // deconz 2.5.54).