search

openclimatefix / graph_weather

PyTorch implementation of Ryan Keisler's 2022 "Forecasting Global Weather with Graph Neural Networks" paper (https://arxiv.org/abs/2202.07575)
MIT License
188 stars 47 forks source link

Add support for Anemoi Datasets #102

Open jacobbieker opened 5 months ago

jacobbieker commented 5 months ago

ECMWF has now started doing an idea similar to the one in #3 Blog post: https://www.ecmwf.int/en/about/media-centre/aifs-blog/2024/data-driven-regional-modelling

Detailed Description

https://anemoi-datasets.readthedocs.io/en/latest/using/combining.html

Context

Using a high-resolution subset within a global model is quite interesting to get the higher resolution required. Anemoi datasets works to do that as well, making the cutouts.

Possible Implementation

Import the package and use that? Copy over the cutout functionality? As otherwise, its a light wrapper around Zarr stores, which we already support.

0xFrama commented 5 months ago

Hi @jacobbieker, could you give me more details about this?

jacobbieker commented 5 months ago

Hi, we just want to make it so we can train with the anemoi datasets package. It has some nice functionality, so would want to add it to the different ways of loading weather datasets. Probably as another option in the data/ module here, somewhat like the IFS dataloader.

0xFrama commented 5 months ago

Have you a specific weather dataset in mind to start with?

jacobbieker commented 5 months ago

The easiest would probably be either WeatherBench, or get it working with the ICON archive that we've made on Hugging Face.

With the ICON EU and ICON Global datasets on HF, we could get the loading of a low resolution global model + high resolution regional model working together, but would take more work to get up and running with this.

First step would probably try WeatherBench, and just get it to output training examples

glitch401 commented 3 months ago

hey, is this issue still being worked on?

jacobbieker commented 3 months ago

I'm not sure, but if you want to tackle it, that would be great!

glitch401 commented 3 months ago

alrighty, taking a jab at it :D