Separate data source for forecasts and measurements

[Flix6x]

I'd like forecasts and measurements to be saved under a separate DataSource (type), so we can treat them separately in e.g. the FlexMeasures UI.

I propose to use the following. For measurements:

Source(
    name="OpenWeatherMap",
    model="onecall/timemachine",  # name of the API, see https://openweathermap.org/api/one-call-api#history
    version="2.5",
    type="archive",  # alternatives: "historian" or "logger"
)

And for forecasts:

Source(
    name="OpenWeatherMap",
    model="onecall"  # name of the API, see https://openweathermap.org/api/one-call-api#current
    version="2.5",
    type="forecaster",
)

Two notes:

• FlexMeasures could use a database migration to fill in the model and version (to this date, this plugin only used the onecall endpoint and not the onecall/timemachine endpoint). Even though such a migration would be specific to this plugin, I reckon it should be part of the db versioning of FlexMeasures.
• As a related issue, I'm considering using a db migration to rename the "forecasting script" data source type to "forecaster" in FlexMeasures.
• OWM's latest API version is actually 3, see https://openweathermap.org/api/one-call-3, so let's consider updating (in a separate issue).

[Ahmad-Wahid]

As much I understood, is that we are going to use 2 endpoints such that one for current forecasts and other for history.??

[Flix6x]

As much I understood, is that we are going to use 2 endpoints such that one for current forecasts and other for history.??

Correct.

[nhoening]

I see some problems in the way of this issue:

• Actually, we only have one command here which gets forecasts, and we don't support historical forecasts yet. Wouldn't this issue then require that we get a new command, or add --start-day and --end-day to the current one?
• More importantly, OWM only offers "previous 5 days", and other historical data can only be gotten in their "Historical weather collection" packages, or with the "history" API which is very expensive, see https://openweathermap.org/price. Thus, we can get by with what we have now, and add historical data for 9$ a location if needed. For that, we could indeed write an import script and use a different data source, but I would do that when we get to that point.

@Flix6x any thoughts?

[Flix6x]

We're using the One Call API, right? It features historical data, too.

[nhoening]

I now found that there is indeed a dt parameter possible, but somewhere I read the hint about the "previous 5 days". In any case,

• we'd need CLI structure updates, as well (new parameters), so this issue is not ready.
• we are not sure if we need this, because as it stands, we either go live with a location, or we buy the archive.

[Flix6x]

But going live with a location should mean, imo, that we query both recent forecasts and recent measurements, with 2 separately API calls. Possibly within one CLI call, or two, I don't mind.

[Flix6x]

The status quo is that this plugin is not collecting data that FlexMeasures interprets as measurements. The reason for that is that FM switched from distinguishing forecasts by horizon, to distinguishing forecasts by source type.

[Ahmad-Wahid]

I now found that there is indeed a dt parameter possible, but somewhere I read the hint about the "previous 5 days". In any case,

• we'd need CLI structure updates, as well (new parameters), so this issue is not ready.
• we are not sure if we need this, because as it stands, we either go live with a location, or we buy the archive.

you're right, this call is limited to only 5 days. I have an idea about this, what if we try to increment the time (+5 days) and get the historical data in multiple calls?

[nhoening]

Okay, so I finally caught up on what @Flix6x actually wants and I have an idea what this issue might do - I think.

Next to currently available forecasts, we want to also collect one real measurement per time step (hour) and weather sensor. And we want to label that with a different data source.

We definitely need a different data source (mostly work in utils/modeling.py).

How to collect a measurement per hour?

Simple approach: Each API call gives us the current weather. We could just use that. Of course that works well if you call the OWM API around the full hour, but of course we can save (ans use) any datetime. All this would need might be a parameter --save-current for the CLI command.

Complex approach: Add a second command which accepts a timestamp from the past and looks up exactly that via the timemachine API endpoint (using only_current = True).

Mixed approach: Do not add a second command, but do call the OWM API twice (once for forecasts, once for an exact measurement on the recent hour).

[Flix6x]

This issue needs a tech spike imo.

A possible problem with using the current weather from the OWM forecast endpoint is that it is likely still a forecast, and, more importantly, represents 1-minute averages, not the most recent 1-hour averages, leading to wrong interpretations of the data (see https://openweathermap.org/accuracy-and-quality, which is the only place I found with a hint about what the current data actually represents).

Furthermore, note that:

-One Call API version 2.5 restricts historical data to the last 5 says: see https://openweathermap.org/api/one-call-api

• One Call API version 3 no longer has this restriction: see https://openweathermap.org/api/one-call-3

[Ahmad-Wahid]

if version 3 has no restriction, then I believe we can call it twice for the two different data sources(forecasts, and measurements)