Eve history

[simont77]

Hi, I made some progress with Eve history (see my gist) looking at the communication of a real Eve Room, but now I would need to monitor also other accessories. Do you have any possibilities? Simone

[simont77]

Hi again, now I've definitely understood Eve history protocol, at least enough to create a fake accessory whose history will show up in Eve.app. Unfortunately I'm not very good with nodejs, so translating it in a full featured history plugin for homebridge would take some time. Are you more skilled than me? I think that maybe the best way to proceed is to develop a library that can be included in any plugin. You will find all my latest findings in my gist, let me know if you need some clarification. Simone

[skrollme]

Hi @simont77, that are great news. I took a quick look at your updated gist. I think it will not be easy to implement this in a homebridge plugin, especially because I'm not a pro at NodeJS, too ;) But I promise I will try it soon and let you know.

[simont77]

Good! I will try as well, I'm starting from your eveatmo-history module. In the meantime I updated my gist with a tentative communication protocol, just to clean up a bit all the stuff.

[simont77]

This is a rough code, not so elegant probably, but it works, at least for testing. TODO: a lot! Implement the actual history (maybe in a persistent way), add a method to create the entries (for different accessories), deal with timestamps and memory rolling.

class S2R1Characteristic extends Characteristic {
    constructor() {
        super('S2R1', 'E863F116-079E-48FF-8F27-9C2605A29F52');
        this.setProps({
            format: Characteristic.Formats.DATA,
            perms: [
                Characteristic.Perms.READ, Characteristic.Perms.NOTIFY
            ]
        });
    }
}

class S2R2Characteristic extends Characteristic {
    constructor() {
        super('S2R2', 'E863F117-079E-48FF-8F27-9C2605A29F52');
        this.setProps({
            format: Characteristic.Formats.DATA,
            perms: [
                Characteristic.Perms.READ, Characteristic.Perms.NOTIFY
            ]
        });
    }
}

class S2W1Characteristic extends Characteristic {
    constructor() {
        super('S2W1', 'E863F11C-079E-48FF-8F27-9C2605A29F52');
        this.setProps({
            format: Characteristic.Formats.DATA,
            perms: [
                Characteristic.Perms.WRITE
            ]
        });
    }
}

class S2W2Characteristic extends Characteristic {
    constructor() {
        super('S2W2', 'E863F121-079E-48FF-8F27-9C2605A29F52');
        this.setProps({
            format: Characteristic.Formats.DATA,
            perms: [
                Characteristic.Perms.WRITE
            ]
        });
    }
}

class FakeGatoHistoryService extends homebridge.hap.Service {
    constructor() {
        super("History", 'E863F007-079E-48FF-8F27-9C2605A29F52');

        this.lastEntry = 1;
        this.history = [];
        this.maxHistory = 100;
        this.currentEntry = 0;
        this.transfer=false;
        this.setTime=true;

        this.addCharacteristic(S2R1Characteristic);

        this.addCharacteristic(S2R2Characteristic)
            .on('get', (callback) => {
                if ((this.currentEntry<=this.lastEntry) && (this.transfer==true))
                {

                    if (this.setTime==true)
                    {   
                        console.log("Data: "+ "15" + numToHex(swap16(this.currentEntry)).toString('hex') + "0000 0801 0000 81 b093 331f 0000 0000 00 0000");
                        callback(null,hexToBase64('15' + numToHex(swap16(this.currentEntry),4) +' 0000 0801 0000 81 b093 331f 0000 0000 00 0000'));
                        this.setTime=false;
                    }
                    else
                    {   
                        console.log("Data: "+ "10 " + numToHex(swap16(this.currentEntry)).toString('hex') + " 0000 " +numToHex(swap16((this.currentEntry))).toString('hex') + " 0000 07 a60b 9c15 1302");
                        callback(null,hexToBase64('10' + numToHex(swap16(this.currentEntry),4)+ ' 0000 ' + numToHex(swap16(this.currentEntry),4) + ' 0000 07 a60b 9c15 1302'));
                    }
                    this.currentEntry++;
                }
                else
                    this.transfer=false;

        }); 

        this.addCharacteristic(S2W1Characteristic)
            .on('set', this.setCurrentS2W1.bind(this));

        this.addCharacteristic(S2W2Characteristic)
            .on('set', this.setCurrentS2W2.bind(this));

    }

    sendHistory(address){
        var hexAddress= address.toString('16');
        if (address!=0)
            this.currentEntry = address;
        else
            this.currentEntry = 0;
    }

    addEntry(entry){
        if (this.lastEntry<this.maxHistory)
        {
            this.history[this.lastEntry] = (entry);
            this.lastEntry++;
        }
        else
        {
            this.history[0] = (entry);
            this.lastEntry = 1;
        }

        this.getCharacteristic(S2R1Characteristic)
            .setValue(hexToBase64('3dd0000008010000b093331f040102020203020f03' + numToHex(swap16(this.lastEntry)) +'ed0f00000000000000000101'));
        console.log("Last entry: "+ this.lastEntry.toString(16));

    }

    setCurrentS2W1(val, callback) {
        callback(null,val);
        console.log("Data request: "+ base64ToHex(val));
        var valHex = base64ToHex(val);
        var substring = valHex.substring(4,12);
        var valInt = parseInt(substring,16);
        var address = swap32(valInt);
        var hexAddress= address.toString('16');

        console.log("Address requested: "+ hexAddress);
        if (this.transfer==false)
        {
            this.transfer=true;
            this.sendHistory(address);
        }
    }

    setCurrentS2W2(val, callback) {
        console.log("Clock adjust: "+ base64ToHex(val));
        callback(null,val);
    }

}

[skrollme]

Awesome! I will take this as a base for further development..

[simont77]

This is more functional, it accepts real EveWeather-styled logs with time stamps. https://gist.github.com/simont77/919399dda338e37213b1ca231082c219

Simply add new entries with something like:

this.loggingService.addEntry({time: moment().unix(), temp:this.temperature, pressure:this.airPressure, humidity:this.humidity});

Actually data download is far faster than with real accessory. Still to implement memory rolling, persistent memory and more accessories. Probably from time to time it will be necessary to adjust the reference time stamp for long running uses (now it is set at the first sample after program launch), and for sure with this protocol is not possible to address more the 2^16 entries (supposing that Eve.app would not stop well before, since real accessory are stated for 2-3 weeks of data every 10 minutes, i.e. about 3000 entries). And bug fixes, of course

[simont77]

I set up a proper repository for the fakegato-history service https://github.com/simont77/fakegato-history

[skrollme]

Thanks, I will look into it. Before continuing with the history service I will refactor the plugin a little bit. Especially the way the API is called. There are too many unnecessary Calls when checking the devices in a Homekit app.

[oynek]

Would be great to see this implemented! Great work btw.

[oynek]

Any progress on this?

[DJay-X]

I think there is a breakthrough with recording to the history by the plugin of @naofireblade https://github.com/naofireblade/homebridge-weather-station-extended You guys should definitely use the synergy. Awesome work that you all are doing. Thanks for sharing.

@skrollme You are warmly invited to come to our Forum on SmartApfel.de if you are not already there incognito. ;)

[naofireblade]

I use the awesome fakegato-history from @simont77 in my plugin and it's working smooth so far :) .

[skrollme]

Thanks for all your input. I recent times I have some other projects but I promise I will continue with this

[skrollme]

"Good news everyone!" Thanks to the great and almost drop-in-and-it-works fakegato-history module by @simont77 I got the history working for homebridge-eveatmo.

At the moment it only works for for temperature (indoor/outdoor), humidity (indoor/outdoor) and pressure (outdoor). The history for the CO2-Sensor does not work so far although I add the co2 values, but I will check this soon.

Since yesterday evening the current state of the plugin is working for me (https://www.dropbox.com/s/ysg16pb0ajnc318/IMG_5712.PNG?dl=0) and after some cleanup and refactoring (and a little more testing) I will publish this as a new version of the plugin.

[simont77]

Unfortunately you can use history only for services supported by Eve, and only once I have decoded the protocol for that particular sensor. So, no history for CO2 for the time being. Instead you can use the Eve custom service E863F10B for PPM air quality, but it doesn't seem to be strictly related to CO2 https://help.elgato.com/customer/en/portal/articles/2233021-eve-room-—-does-it-detect-co2-co₂-carbon-dioxide-, it is something "equivalent": just try and see what happens

[skrollme]

Is this the "AirQuality"-Characteristic which the indoor/room-Devices show? These seem to support a history in the Eve.app (currently showing an empty graph). Does your module already support this kind of history-type?

[simont77]

Yes, you should use type “room”, for temp, humidity and PPM history

[skrollme]

Hmm.. That is exactly what I have already done. See:

Service creation: https://github.com/skrollme/homebridge-eveatmo/blob/540a64a0a57f9d5fa3a8d0afa033a82c1eb56fcd/accessory/eveatmo-room-accessory.js#L69

And addEntry: https://github.com/skrollme/homebridge-eveatmo/blob/540a64a0a57f9d5fa3a8d0afa033a82c1eb56fcd/accessory/eveatmo-room-accessory.js#L95

At my local machine I also tried to remove the extra characteristics from the air-quality-service (https://github.com/skrollme/homebridge-eveatmo/blob/history_by_fakegato/service/eveatmo-room-airquality.js) but that did not make a difference. Do you have any clue what to change here, @simont77 ?

[simont77]

try to add characteristic 132 under the air quality service, to better match the original one https://github.com/simont77/fakegato-history/wiki/Services-and-characteristics-for-Elgato-Eve-devices#elgato-eve-room-firmware-revision-131466

[simont77]

From my old notes on first Eve Room experiment: “In order to clone the Eve Room I had to reproduce the two custom characteristics for air quality and put them inside the standard air quality service (as in real Eve), otherwise air quality logs are not seen by Eve.app.”

These are 10B and 132, 10B is PPM, 132 unknown but apparently required.

[skrollme]

That seems to work but I initially had some problems that only one "room" (Netatmo's extra indoor sensors) shows an co2/airquality-history. The other "room" (Netatmo base station) did not show the history although this are the exact same services,characteristics and even node-classes.

Another point which I'm not sure about is, that the "room" accessories show...

• Airquality (which display an co2-ppm graph)
• Temperature (with temp-graph)
• Humidity (with hum-graph)
• AND an extra co2 characteristic without an graph

I will check this in the HMCatalog.app tomorrow

[skrollme]

The AirQuality history seems to work now, too. 🎉 I will merge the feature and create a new npm-version from this.

[skrollme]

Version 0.3.0 with history-support is out and published to npm.

[DJay-X]

Top! 👍🏻 Thanks a lot. And looks very good at first glance. Works! I would only be interested in what is meant by...

Polling/caching still needs some more finetuning. Please let me know if you see to many "Loading new data from API..." log-entries.

What would be "to many"? In my log, data is retrieved from the API every 15 seconds. "to many"? 🤔😉

Feb 15 21:37:29 raspberrypi homebridge[517]: [2018-2-15 21:37:29] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:37:44 raspberrypi homebridge[517]: [2018-2-15 21:37:44] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:37:59 raspberrypi homebridge[517]: [2018-2-15 21:37:59] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:38:14 raspberrypi homebridge[517]: [2018-2-15 21:38:14] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:38:29 raspberrypi homebridge[517]: [2018-2-15 21:38:29] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:38:44 raspberrypi homebridge[517]: [2018-2-15 21:38:44] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:38:59 raspberrypi homebridge[517]: [2018-2-15 21:38:59] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:39:14 raspberrypi homebridge[517]: [2018-2-15 21:39:14] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:39:29 raspberrypi homebridge[517]: [2018-2-15 21:39:29] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:39:44 raspberrypi homebridge[517]: [2018-2-15 21:39:44] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:39:59 raspberrypi homebridge[517]: [2018-2-15 21:39:59] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:40:14 raspberrypi homebridge[517]: [2018-2-15 21:40:14] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:40:29 raspberrypi homebridge[517]: [2018-2-15 21:40:29] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:40:44 raspberrypi homebridge[517]: [2018-2-15 21:40:44] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:41:00 raspberrypi homebridge[517]: [2018-2-15 21:40:59] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:41:15 raspberrypi homebridge[517]: [2018-2-15 21:41:15] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:41:30 raspberrypi homebridge[517]: [2018-2-15 21:41:30] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:41:45 raspberrypi homebridge[517]: [2018-2-15 21:41:45] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:42:00 raspberrypi homebridge[517]: [2018-2-15 21:42:00] [eveatmo platform] Loading new data from API for: weatherstation Feb 15 21:42:15 raspberrypi homebridge[517]: [2018-2-15 21:42:15] [eveatmo platform] Loading new data from API for: weatherstation

But so far I don't notice any problems. Although I don't think 15 seconds are really necessary. Any minute would do the same. At least with me. Polling configurable via config. json would be fantastic. Then everyone could adjust individually what is needed.

Thanks again for your great work and sharing @skrollme.

[DJay-X]

I noticed I still had the ttl 15 line in the config.json My bad. 😬 I removed the complete line and will check the results in history tomorrow.

[skrollme]

😄 Yap, 15s is a bit to frequent. Keep in mind that the Netatmo stations record a value every 5minutes so values far below 5min does not make any sense at all. Maybe I should add a interval = max(interval,5*60) to prevent this "overpolling"

[r32er]

With my Netatmo outdoor and addition Modul the history works fine except for a few gaps. But the base module hasn't written a history since this morning.

[skrollme]

@r32er currently not sure about this. I have neither gaps nor a module with no history at all. Can you take a look into your ~/.homebridge directory: There should be some persistence-json-files, one for each module. They are named like "YOURPIE'SHOSTNAME_NETATMOMODULENAME.json"

• Do you have these files?
• Have you tried completely quitting the eve.app and/or pull to refresh inside the app?

[r32er]

I checked and didn't have those files. But the external module and the additional module record the history. Except the base module...

Yes I tried to restart the app and refresh it.

Maybe it helps to reinstall the plugin.

[skrollme]

So maybe the files are placed anywhere else. At least I did not change the file location in the fakegato-init. Of course you can try a reinstall. Let me know if this makes a difference for you. I just published a small update but this should only affect the CO2-Sensors updates.

[r32er]

Unfortunately, reinstalling the plugin did not help.

The history in my base modul works till yesterday 08:19. Then i stopped the homebridge to change the ttl config to 600 to avoid the overpolling. default was 15.

Since then the base modul don't save the history :(

But the files should be there, in which instance they are loaded.

[skrollme]

Once after restarting homebridge I saw that there was a gap in the history back untill the time the eve.app last synced. But that was the only problem so far for me. Did you started homebridge with different users and/or permissions? Maybe the history files are not writeable/readable now.

I also published a new version (0.3.2) with a minimum polling interval of once every 5min.

[r32er]

I have reinstalled the eve app, eveatmo Plugin. Restart the Homebridge and pi. Delete the accessories und persist folder. But nothing helped.

The base Modul don’t write the history :( And I can’t find the files were the history is written in.

Homebridge run as systemd. And eveatmo run in an second instance.

[r32er]

I found the files at ~/root/homebridge. why are the files stored in root at all? Do you have an idea?

The last access was yesterday at 09:52 at the file which is my base modul. the file is empty.

[skrollme]

The storage location is chosen by the fakegato-history plugin itself. See: https://github.com/simont77/fakegato-history#file-system In my setup homebridge is running via systems as well, but the files are placed inside ~/.homebridge.

But back to the problem: It is strange that the basestation does not write history at all. Did you changed the file-permissions for the persist-files (try 777)? Does the history works if you run homebridge manually (without systemd)?

[r32er]

i have done: root@homebridge2:/home/pi# sudo chmod 777 /root/.homebridge/homebridge2_Home\ Wohnzimmer_persist.json

is that right? dont help. the file is still empty.

I have deleted the file in the hope that a new one will be created. Unfortunately, no new ones are created.

i search for the fakegato config but i can't find it... maybe there i can set up something?

[skrollme]

Basically that was what I meant, yes. Fakegato is not configured by you but by the plugin which uses it (in this case homebridge-eveatmo). Maybe @simont77 has an idea why only one device fails here. Some bad implementation / integration in homebridge-eveamto?

[simont77]

For file system persistence, fakegato saves files into the default homebridge folder (.homebridge in the home folder of the user launched homebridge) or the folder defined by the user with the -U option of homebridge. If you are using systemd, you are probably following this guide https://gist.github.com/johannrichard/0ad0de1feb6adb9eb61a/, which instructs you create a folder owned by a newly created homebridge user so that the -U option of the startup script can use it. Shouldn't be root folder, I suspect this may cause problems. So your data should appear in /var/lib/homebridge, together with the usual accessories/ and persist/ folders

[r32er]

I have different Homebridge instances. I had expected the persist files where the history is written in the Homebridge folder of the Homebridge instance in which the plugin is configured (config.json)

I wonder why the the files are in /root/.homebridge.

[Nastras]

Hello everybody,

with me the data writing works. The responsible files are also in / root / homebridge.

Also the fakegato files of homebridge-hue.

Should it be like this?

[Nastras]

[simont77]

wait, apparently by default fakegato saves history in os.homedir(),'.homebridge'. I though we already switched that to homebridge user folder if defined. This would explain why it is saving in root folder i you are running homebridge as root. I will patch fakegato asap, however with the right privileges on the root folder it should not crete any issue.

[r32er]

it seemls like a bug for me. yesterday i got the history for the basemodul in living room but then the outdoor modul history dont work. an actuallly only my bedroom works..

[skrollme]

A few days ago I had one gap as well. It seemed that I did not fetch the data for too long, so the persistence-file could not keep all entries. I increased the size for the persistence-files for testing and everything seems to work fine.

I finalize this a will publish an update to npm.js this evening.

For now I will close this Issue here because the history in general is working now.