ecmwf / anemoi-training

Apache License 2.0
17 stars 17 forks source link

Allow noncontinous time ranges in datasets for training and validation #128

Open HCookie opened 1 week ago

HCookie commented 1 week ago

Is your feature request related to a problem? Please describe.

Currently training, validation and test datasets are continuous time ranges. While this makes sense, it may be interesting to explore other ways of organising the time slices.

Describe the solution you'd like

Add a way to provide ranges of time to the dataloader, allowing the following config.

training:
  dataset: ${dataloader.dataset}
  ranges: 
    - [1970, 1980]
    - [1990, 2020]
  frequency: ${data.frequency}
  drop:  []

validation:
  dataset: ${dataloader.dataset}
  ranges: 
    - [1981, 1989]
    - [2021, 2021]
  frequency: ${data.frequency}
  drop:  []

Describe alternatives you've considered

No response

Additional context

No response

Organisation

ECMWF