Unexpected time step detected in historic data

[raoulbhatia]

After #193 was fixed (QUARTER_HOUR vs. DAY granularity), I am one step further in importing historical data.

Judging from the logs, wnsm is able to properly fetch the historical data, but then fails to import it with the following warnings:

2023-10-29 18:12:05.079 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Starting import of historical data. This might take some time.
2023-10-29 18:12:05.994 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Unexpected time step detected in historic data: {'messwert': 2179.0, 'zeitVon': '2023-05-29T22:00:00.000Z', 'zeitBis': '2023-05-30T22:00:00.000Z', 'qualitaet': 'VAL'}
2023-10-29 18:12:05.995 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Unexpected time step detected in historic data: {'messwert': 4476.0, 'zeitVon': '2023-05-30T22:00:00.000Z', 'zeitBis': '2023-05-31T22:00:00.000Z', 'qualitaet': 'VAL'}
2023-10-29 18:12:05.996 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Unexpected time step detected in historic data: {'messwert': 7459.0, 'zeitVon': '2023-05-31T22:00:00.000Z', 'zeitBis': '2023-06-01T22:00:00.000Z', 'qualitaet': 'VAL'}
2023-10-29 18:12:05.997 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Unexpected time step detected in historic data: {'messwert': 4776.0, 'zeitVon': '2023-06-01T22:00:00.000Z', 'zeitBis': '2023-06-02T22:00:00.000Z', 'qualitaet': 'VAL'}
2023-10-29 18:12:05.997 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Unexpected time step detected in historic data: {'messwert': 5258.0, 'zeitVon': '2023-06-02T22:00:00.000Z', 'zeitBis': '2023-06-03T22:00:00.000Z', 'qualitaet': 'VAL'}
2023-10-29 18:12:05.998 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Unexpected time step detected in historic data: {'messwert': 3896.0, 'zeitVon': '2023-06-03T22:00:00.000Z', 'zeitBis': '2023-06-04T22:00:00.000Z', 'qualitaet': 'VAL'}
[...]
2023-10-29 18:12:06.113 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Unexpected time step detected in historic data: {'messwert': 10244.0,'zeitVon': '2023-10-27T22:00:00.000Z', 'zeitBis': '2023-10-28T22:00:00.000Z', 'qualitaet': 'VAL'}

Note: The integration was enabled 2023-10-09. The first data appeared 2023-10-10 10.00h CEST.

My current observations:

1. Historical data is not (yet) imported, See Appendix A
2. Data seems to be reported from 00:00 UTC to 00:00 UTC the next day, see the logs above
3. In the dashboard, the data shows at "arbitrary" times – presumably at the time the data was made available via the API. See the screenshots attached as well, Appendix B
4. There seem to be some hard-coded references to "15 minutes" in the code – perhaps this interferes with the daily reporting that is the default for Smart Meters in Austria? See https://github.com/DarwinsBuddy/WienerNetzeSmartmeter/blob/main/custom_components/wnsm/statistics_sensor.py#L156-L162 as of 2023-10-29

Cheers, Raoul

Appendix A Missing Historical Data

Appendix B Different Import Times

[raoulbhatia]

While clicking through the homeassistant UI, trying to add cost to my Electricity grid Energy Dashboard, 2 more WARNINGs appeared:

2023-10-29 19:12:10.552 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Data starting at 2023-10-28 23:00:00+00:00 does not contain granular data! Opt-in was not set back then.
2023-10-29 19:12:11.323 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Data starting at 2023-10-29 23:00:00+00:00 does not contain granular data! Opt-in was not set back then.

[reox]

There seem to be some hard-coded references to "15 minutes" in the code – perhaps this interferes with the daily reporting that is the default for Smart Meters in Austria? See https://github.com/DarwinsBuddy/WienerNetzeSmartmeter/blob/main/custom_components/wnsm/statistics_sensor.py#L156-L162 as of 2023-10-29

Yes, this was intentional, as I had no idea how daily data would look like. Now we know :) And the other thing is, that AFAIR daily data cannot really be imported into the statistics, but only hourly data. One cheap trick would be to divide the value by 24 and import all of them...

[raoulbhatia]

Personally, I'd be OK a h that ie better to have some data with caveats compared to no data.

Though I am confused as to why import of historic data would be any different to daily data that is newly ingested.

Perhaps this should be made a configurable option? ie also the daily data could be the imported as /24 to get hourly data?

If dixumentrd well, this might be less confusing for the user?