ECMWF cumulative variables

openclimatefix / ocf_datapipes

OCF's DataPipe based dataloader for training and inference

MIT License

13 stars 11 forks source link

ECMWF cumulative variables #286

Closed peterdudfield closed 1 month ago

peterdudfield commented 4 months ago

There are some variables that are cumulative.

This is quite tricky for the model to handle, as delayed in data can have a real impact. OF course the model should learn to take the difference but, it perhaps is easier to give the model what we want

Tasks

add datapipes that adds/replaces variables with the diff
[ optional] Adjust ECMWF normalisation values for diffed variables
Retrain PVnet UK with these diffs
Retrain Windnet with these diffs

jacobbieker commented 4 months ago

It seems like wind variables are not cumulative, so it shouldn't affect windnet too much, but definetly should retrain PVNet for india with it. And do diffs for all cumulative variables in the future, as I don't think any model will handle it very well, and normalization won't work very well for this

dfulu commented 1 month ago

This is solved by #314. The only thing not covered would be to add normalisation values for the diffed ECMWF variables over India

peterdudfield commented 1 month ago

Shall i close this now @dfulu ?

dfulu commented 1 month ago

Shall i close this now @dfulu ?

Yeh sure