Adding historical data from a cloud backend

[faune]

Hi, I have a sensor which store data in a cloud solution. In a homebridge plugin using fakegato I have two issues:

1. The cloud contain historical Sensor data. How do I add this to Fakegato? Using addEntry does not work, presumably because that api expects an entry every 10 minutes time stamp (?)

2. Same as above, sensor only report data every 24 hours. How can I add it?

[simont77]

Which kind of sensor is it? can you share a snippet of your code? For some type of sensors, Eve expects a data every 10 minutes, however you can use fakegato timer to average more frequent samples (for instance for temperature). If too sparse samples are provided, you should see holes in the history, but data should appear in any case. In other cases, e.g. for door, extra entries are automatically added every 10 minutes by fakegato repeating the last state.

[faune]

I have historical sensor data going back the last year stored in a cloud service as timestamp, temperature, and humidity. I would like to add this type of data to Fakegato, but it seems addEntry() is not the correct way to do this.

Edit: The code in question is here: https://github.com/faune/homebridge-grohe-sense although I have not yet pushed the feature branch where I am working with fakegato.

[simont77]

Please provide only the relevant code, so that we can check. AddEntry() is the only way to add data to history, however data should be properly prepared. In particular check the time stamps, if they are in the future with respect to the local time the history will be populated, but no data will be shown.

[faune]

My question is: can addData() be used to add historical data as well, where time stamp is E.g. 1 year or 1 month or 1 week in the past? Because this is what I can’t get working. I have adjusted epoc so 01.01.01 is deducted from the time stamp, but When using the eve app no old data is included in the history.

[simont77]

The first AddEntry that you call set the beginning of time for Eve. Then the following AddEntry have to follow the initial one. So, it should be possible to add data 1 year in the past, provided you respect these rules. If you already messed the history in Eve, you may try starting over your plugin and local history (if you persisted it) and doing an AddEnty with current time, which may reset the reference time for Eve.

[faune]

Thats explain my issues I think, because I had two async functions where one fetched historical data and one fetched current data, and both call addEntry(). If the current data is added before the historical data, then it won't work according to your explanation.

How exactly do I reset the Eve history? Is it enough to delete the cached accessory in Homebridge?

[faune]

Code that uses addEntry:

getHistoricalMeasurements() {
    this.ondusPlatform.log.debug(`[${this.logPrefix}] Fetching historical temperature and humidity levels`);

    // Fetch all registered appliance measurements from last month up until yesterday
    // If no measurements are present, the following is returned {"code":404,"message":"Not found"}
    const lastMonthDate = new Date();
    const yesterdayDate = new Date();
    lastMonthDate.setDate(lastMonthDate.getDate() - 30);
    yesterdayDate.setDate(yesterdayDate.getDate() - 1);
    this.getApplianceMeasurements(lastMonthDate, yesterdayDate)
      .then( measurement => {
        const measurementArray = measurement.body.data.measurement;
        if (!Array.isArray(measurementArray)) {
          throw Error(`Unknown response: ${measurementArray}`);
        }

        // Sort historical measurements
        this.ondusPlatform.log.debug(`[${this.logPrefix}] Retrieved ${measurementArray.length} historical measurements`);
        measurementArray.sort((a, b) => {
          const a_ts = new Date(a.timestamp).getTime();
          const b_ts = new Date(b.timestamp).getTime();
          if(a_ts > b_ts) {
            return 1;
          } else if(a_ts < b_ts) {
            return -1;
          } else {
            return 0;
          }
        });
        measurementArray.forEach( value => {
          this.historyService.addEntry({time: moment(value.timestamp).unix() - moment('2001-01-01T00:00:00Z').unix(), 
            temp: value.temperature, humidity: value.humidity});
        });
      })
      .catch( err => {
        this.ondusPlatform.log.error(`[${this.logPrefix}] Unable to retrieve historical temperature and humidity: ${err}`);
      });
  }

A total of 694 entries are fetched from the cloud service and added with this function. In the log I get some suspicious messages:

[9/9/2020, 9:29:10 PM] [Ondus] Clock adjust Sense Teknisk: 86640925
[9/9/2020, 9:29:10 PM] [Ondus] Data request Sense Teknisk: 01ff0300000000
[9/9/2020, 9:29:10 PM] [Ondus] Address requested Sense Teknisk: 3
[9/9/2020, 9:29:11 PM] [Ondus] Clock adjust Sense Teknisk: 87640925
[9/9/2020, 9:29:11 PM] [Ondus] Data request Sense Teknisk: 01ff0300000000
[9/9/2020, 9:29:11 PM] [Ondus] Address requested Sense Teknisk: 3
[9/9/2020, 9:29:12 PM] [Ondus] Clock adjust Sense Teknisk: 88640925
[9/9/2020, 9:29:12 PM] [Ondus] Data request Sense Teknisk: 01ff0300000000
[9/9/2020, 9:29:12 PM] [Ondus] Address requested Sense Teknisk: 3
[9/9/2020, 9:29:13 PM] [Ondus] Clock adjust Sense Teknisk: 88640925
[9/9/2020, 9:29:13 PM] [Ondus] Data request Sense Teknisk: 01ff0300000000
[9/9/2020, 9:29:13 PM] [Ondus] Address requested Sense Teknisk: 3
[9/9/2020, 9:29:14 PM] [Ondus] Clock adjust Sense Teknisk: 8a640925
[9/9/2020, 9:29:15 PM] [Ondus] Data request Sense Teknisk: 01ff0300000000
[9/9/2020, 9:29:15 PM] [Ondus] Address requested Sense Teknisk: 3
[9/9/2020, 9:29:16 PM] [Ondus] Clock adjust Sense Teknisk: 8c640925
[9/9/2020, 9:29:17 PM] [Ondus] Data request Sense Teknisk: 01ff0300000000
[9/9/2020, 9:29:17 PM] [Ondus] Address requested Sense Teknisk: 3
[9/9/2020, 9:29:19 PM] [Ondus] Clock adjust Sense Teknisk: 8f640925
[9/9/2020, 9:29:19 PM] [Ondus] Data request Sense Teknisk: 01ff0300000000
[9/9/2020, 9:29:19 PM] [Ondus] Address requested Sense Teknisk: 3
(...)

Still can only see 3 entries in Eve app. Still not sure what the problem is, but hopefully this can give you some clue?

Edit: After clearing the Eve history in the app, and making sure only historical data is added upon plugin initialization it seems the history memory array inside fakegato is still empty. If I keep it running after starting, I get the following in the log:

[9/9/2020, 9:36:41 PM] [Ondus] [Sense Teknisk] Retrieved 694 historical measurements
[9/9/2020, 9:37:41 PM] [Ondus] First entry Sense Teknisk: 0
[9/9/2020, 9:37:41 PM] [Ondus] Last entry Sense Teknisk: 2
[9/9/2020, 9:37:41 PM] [Ondus] Used memory Sense Teknisk: 2
[9/9/2020, 9:37:41 PM] [Ondus] 116 Sense Teknisk: 00000000000000008566092503 0102 0202 03020300c00f00000000000000000101
[9/9/2020, 9:38:41 PM] [Ondus] First entry Sense Teknisk: 0
[9/9/2020, 9:38:41 PM] [Ondus] Last entry Sense Teknisk: 3
[9/9/2020, 9:38:41 PM] [Ondus] Used memory Sense Teknisk: 3
[9/9/2020, 9:38:41 PM] [Ondus] 116 Sense Teknisk: 3c000000000000008566092503 0102 0202 03020400c00f00000000000000000101
[9/9/2020, 9:39:41 PM] [Ondus] First entry Sense Teknisk: 0
[9/9/2020, 9:39:41 PM] [Ondus] Last entry Sense Teknisk: 4
[9/9/2020, 9:39:41 PM] [Ondus] Used memory Sense Teknisk: 4
[9/9/2020, 9:39:41 PM] [Ondus] 116 Sense Teknisk: 78000000000000008566092503 0102 0202 03020500c00f00000000000000000101
[9/9/2020, 9:40:41 PM] [Ondus] First entry Sense Teknisk: 0
[9/9/2020, 9:40:41 PM] [Ondus] Last entry Sense Teknisk: 5
[9/9/2020, 9:40:41 PM] [Ondus] Used memory Sense Teknisk: 5
[9/9/2020, 9:40:41 PM] [Ondus] 116 Sense Teknisk: b4000000000000008566092503 0102 0202 03020600c00f00000000000000000101
[9/9/2020, 9:41:41 PM] [Ondus] First entry Sense Teknisk: 0
[9/9/2020, 9:41:41 PM] [Ondus] Last entry Sense Teknisk: 6
[9/9/2020, 9:41:41 PM] [Ondus] Used memory Sense Teknisk: 6
[9/9/2020, 9:41:41 PM] [Ondus] 116 Sense Teknisk: f0000000000000008566092503 0102 0202 03020700c00f00000000000000000101

This seems to indicate that the 694 historical entries are not successfully added, however I am not able to figure out why.

[simont77]

Why are you subtracting moment('2001-01-01T00:00:00Z') from the timestamp? I suppose this would result in an unusually small timestamp, that may be out of the range accepted by Eve.

Also, the history in fakegato and in Eve are generally not synchronized. Eve has its own database, which you can reset only reinstalling the app, or, if some data appears, deleting all the History in the history detail panel. Fakegato history is inizialized every time you start homebridge, unless you persisted the history (check the readme). Fr testing I suggest you to not persist fakegato history. In the low above, I see that fakegato has an increasing number of entry in his history, from 2 to 6. No data transfer to Eve has occurred.

[faune]

I was under the assumption that Eve relies on time 0 to be 2001-01-01 instead of unix time which counts from 1970-01-01. Is that not correct?

[simont77]

Yes, it's correct, but the conversion is done by fakegato.

[faune]

I believe I have had some success now, at least more data shows up. I had to remove Homebridge from HomeKit and re-pair for the data to show up.

[faune]

Dug a little more into my problem, and I have discovered that if I add more than 30 (or perhaps 29) days of data during startup of homebridge, then Eve app will misbehave (resembling an overflow of its internal history), such that only the last 1 or 2 days of data is shown.

I have adjusted the code above to fetch data from range <today-29days, today> and that is at least working. But whenever I try to add data during Homebridge startup that is further back in time than 30 days Eve app either shows nothing or show yesterdays and todays data only. Anyone have experience with this behavior, and can comment?

Edit: Ignore the above, it seems Eve is very picky if there is already history in place for a sensor. After erasing everything, I am able to add all historical data - so success finally!

One last thing I don't understand: In the temperature graph I get two plots following each other quite closely. I was wondering if its the plot from last day/week/month that is overlayed, but that happens when I select "Compare." Then I end with 4 plots in total. This is only one sensor, so I would expect only one plot?

Edit 2: Figured it out, its high and low temperature for the same day. Cheers for your work, although it was quite some work getting this up and running in Typescript :)