Compare ICON weather forecasts to reference

alliander-opensource / AIFES

AI for the Future Energy System

Apache License 2.0

1 stars 1 forks source link

Compare ICON weather forecasts to reference #24

Open FrankKr opened 1 year ago

FrankKr commented 1 year ago

The current backtest notebook uses the most recent Harm Arome weather forecast. This is not really fair if you want to analyze the accuracy for e.g. lead time=24 hours ahead. Let's use the ICON weather forecasts at specific lead times to make a fair assessment of forecast accuracy over lead time.

idea; you can train a specific model for a specific lead time, corresponding to the lead-time of the weather forecasts used in the Predictors.

Acceptance criteria:

[ ] we have a fair dataset to do backtests on
[ ] we know the effect of using weather forecasts with a too short lead time (How much do we overestimate the forecasting performance?)
- especially important with quantiles

MartijnCa commented 1 year ago

I changed the priority of this issue to urgent, because the way we will approach the other research items on the backlog depends on the outcome of this issue.

MartijnCa commented 1 year ago

Hi @woutertromptennet , If I remember correctly, you mentioned that you finished this comparison but that there are some issues with putting the results on GitHub and SharePoint. Have these issues been resolved or is there perhaps anything I could help you with?

woutertromptennet commented 1 year ago

Results were compared, but led to much worse output.

MartijnCa commented 1 year ago

That the results are worse is expected, since weather forecasts with a longer lead time are used. If the results are much worse, we should probably be using the weather forecasts with a longer lead time, so our dataset for the backtest is more fair. In any case it would be nice to have the results of this issue documented.

I am reopening this issue. @SimonVeldkamp, thanks for offering to take up the last mile of this issue.