history data differs from portal

[waldbaer]

Hello,

first I want to say a big thank you for this phantastic library!

Since I integrated this library for logging purposes during this summer I always wondered why the historic data (for a day) always differs from the data reported in the E3/DC portal. Today I played around a little bit and surprisingly found out that the returned values fit much much better if I increment the start timestamp about the time delta between my timezone (Europe/Berlin) and UTC. With such a modified startTimestamp I actually get almost 100% similar values as the portal is reporting.

Currently I unfortunately don't understand why this produces better result, because Europe/Berlin is currently (with daytime savings) 2hrs ahead of UTC. So I would rather expect to subtract the time-delta from the timestamp than adding it.

Did somebody experience the same issue with the history query? Or can someone explain what timestamp (with which timezone etc) is expected by E3/DC?

Some pseudo-code to show what I did:

...
# print day history with integrated library function
history_data = e3dc.get_db_data(timespan="DAY", keepAlive=True);
print("DAY " + formatAsJson(history_data))

# adapted day query with offset in timestamp
requestDate = datetime.datetime.now(tz=timezone('Europe/Berlin')).replace(hour=0, minute=0, second=0, microsecond=0)
startTimestamp = int(time.mktime(requestDate.timetuple()))
startTimestamp += 7200  # time difference between UTC and MESZ in seconds
timespanSeconds = 24*60*60 # 24hrs -> seconds
history_day = e3dc.get_db_data_timestamp(startTimestamp=startTimestamp, timespanSeconds=timespanSeconds, keepAlive=True);
print("DAY_WITH_OFFSET " + formatAsJson(history_data))
...

Here is the difference of the two different ways of history query for today:

"DAY": {
    "autarky": 99.32638549804688,
    "bat_power_in": 9223.625,
    "bat_power_out": 8711.5,
    "consumed_production": 53.85593032836914,
    "consumption": 13376.375,
    "grid_power_in": 11383.75,
    "grid_power_out": 90.10546875,
    "solarProduction": 27849.0,
    "startDate": "2022-10-07",
    "stateOfCharge": 61.47100067138672,
    "timespan": "DAY",
    "timespanSeconds": 86400
  }
"DAY_WITH_OFFSET": {
    "autarky": 99.30683898925781,
    "bat_power_in": 9223.625,
    "bat_power_out": 7382.3125,
    "consumed_production": 51.615272521972656,
    "consumption": 12223.75,
    "grid_power_in": 11379.25,
    "grid_power_out": 84.73046875,
    "startTimestamp": 1665100800,
    "stateOfCharge": 60.32699966430664,
    "solarProduction": 27849.0,
    "timespanSeconds": 86400
  }

and the E3/DC portal reports the following values for today: consumption: 12,21kWh NetIn: 11,38 kWh NetOut: 0,08 kWh Bat (charging): 9,22 kWh Bat (discharging): 7,37 kWh

I would be happy to start a good discussion about this topic here. And of course I am willing to share a potential fix if my experiments actually found an issue.

Best regards Sebastian

[waldbaer]

I did some more research:

My thesis that the start timestamp must be incremented by the time delta to UTC seems to be correct. I did some very small 15min queries and compared them with the single lines of the portal CSV export. The values are matching 100%!

The aggregated sums for a single day seem to be accumulated with a 15min timeshift by the E3DC portal. Similar to exported CSV file the portal is accumulating the 15min blocks starting with 00:00 until 23:45. But this means the very first block (0:00) is the accumulation of 23:45 until 0:00. After substracting 15min from the startTimestamp the outputs are now 100% equal to the portal values. For me this feels like an issue in the E3/DC portal. Does anybody know more about this?

For the MONTH and YEAR queries I also applied the timezone offset. Now the returned values look also 100% similar to the portal data. But: This 15min timeshift issue present for DAY queries seems to be not present for MONTH queries...

Example comparision (library output vs portal, September 2022)

# library output

"history_previous_month": {
    "autarky": 92.2145767211914,
    "bat_power_in": 222936.34375,
    "bat_power_out": 204728.3125,
    "consumed_production": 41.776424407958984,
    "consumption": 359965.6875,
    "grid_power_in": 462624.125,
    "grid_power_out": 28024.865234375,
    "startTimestamp": 1661990400,
    "stateOfCharge": 64.51599884033203,
    "solarProduction": 884434.0,
    "timespanSeconds": 2592000
  }

Values reported by E3/DC portal: Battery Charging: 222,94 kWh Battery Disharging: 204,73 kWh NetIn: 462,62 kWh NetOut: 28.02 kWh Consumption: 359,97 kWh Solar Production: 884,43 kWh

[torbennehmer]

I can confirm that behavior with my S10 unit.

@vchrisb Do you see the same effect on your side as well? I'm especially wondering if all E3DC units run the API in local time mode (Unix Timestams are actually defined as UTC). If so, we should at least document this in some way. For HA, I'll probably do some magic using the tags INFO_REQ_UTC_TIME and INFO_REQ_TIME or maybe INFO_REQ_TIME_ZONE to get the target timezone needed.

[torbennehmer]

Confirmed from my current tests on HA: The Timestamp passed to E3DC needs to be in local time of the unit. Since Python does run the timestamp itself in UTC (correctly), you effectively need to a) create the correct UTC timestamp for your local timezone first and then add the TZ offset a second time to move the timestamp to the apparent local time. If you compare things to the portal, you'll also have to subtract another 15 minutes, as noted above, that seems to be a bug in the E3DC portal which aggregates on 15 minutes intervals and is possible off-by-one somewhere on these intervals.

[torbennehmer]

@vchrisb I am just wondering, would it make sense to add an optional wrapper to the library that works around this and gives the actual real data?