search

vividfog / nordpool-predict-fi

A Python app and ML model that predicts spot prices for the Nordpool FI market.
https://sahkovatkain.web.app/
MIT License
72 stars 8 forks source link

Timestamps off by one hour? #21

Closed ToniA closed 1 month ago

ToniA commented 1 month ago

Looking at the first chart of https://sahkovatkain.web.app/, it seems that the whole time axis is off by one hour. Even the timestamps on the prediction.json seem to suffer from the same problem.

For example, 11.10.2024, at 18:00

vividfog commented 1 month ago

Good catch. I did some preliminary inspection. A proper fix needs changes in Python and Javascript. Made notes, but I'll have to find more focus time to fix it and then also test it, so that browser JS code and Home Assistant clients both see the time zones and DST the same way and YAML updates wouldn't be needed in HA client side.

vividfog commented 1 month ago

Did some more inspection and logging. The Python side time stamp management was correct in the end, even though it's hard to follow due to many different data sources, training/prediction steps, and time zone manipulations required for each. The code could use refactoring for centralized time format/zone handling.

Still in the end, the prediction and Nordpool data, they both end up UTC-timestamped, starting from Helsinki time midnight hour, which is the desired outcome. I don't see errors there. Should double check after DST is over.

For visualizing the data, Javascript fetch() should assume UTC by default, and new Date() should automatically apply time zone conversion to match the user browser's time zone, including DST settings. Explicit time zone setting should not be needed there.

I did find one potential off-by-one issue: When aligning the Sähkötin Nordpool prices and the predictions on the graph with overlapping data shown for "today", the Nordpool data includes one extra hour, 00:00 to 01:00 for "tomorrow". That extra hour should be accounted for, to find where "today" begins and the visual overlap should start. That issue is now fixed.

There is now a lot more logging in the browser developer console. It displays the time stamps from Sähkötin and from the prediction file, in local browser time, and the associated price.

[Log] Sähkötin timestamp: Mon Oct 14 2024 00:00:00 GMT+0300 (Eastern European Summer Time), Price: 3.14 ¢/kWh (scripts.js, line 86)
...
[Log] Prediction timestamp: Mon Oct 14 2024 00:00:00 GMT+0300 (Eastern European Summer Time), Price: 2.008692741394043 ¢/kWh (scripts.js, line 109)

I can't find off-by-ones in yesterday's, today's and tomorrow's prices now. Can you? Please check the browser console log, if you see any discrepancies. The text log is explicit about each hour and associated price, for Nordpool and for predictions.

I'll keep this open for a day or two in case you find something. @ToniA

ToniA commented 1 month ago

Thanks, I think it's fixed now. The chart looks correct to me, as well as the timestamp data on the prediction.json file.