Open HCookie opened 1 week ago
Let's have a look at https://anemoi-datasets.readthedocs.io/en/latest/using/missing.html and have a chat.
How will you handle the last date of the first interval?
You do not want to train using x_i = 1980.12.31 18:00
and x_i+1 = 1990.01.01 00:00
.
In the config, it may turn into this :
training:
concat:
- dataset: ${dataloader.dataset}
start: 1970
end: 1980
- dataset: ${dataloader.dataset}
start: 1990
end: 2020
frequency: ${data.frequency}
how_to_handle_the_date_1980_12_31: 'raise'/'skip'...
Requires https://github.com/ecmwf/anemoi-datasets/pull/118
Allows
Closes #128