Challenge 21 -Diffusion Models on WeatherBench

[EsperanzaCuartero]

Challenge 21 - Diffusion Models on WeatherBench

Stream 2 - Machine Learning for Earth Science

Goal

Develop code and models for diffusion models to predict weather patterns on WeatherBench.

Mentors and skills

• Mentors: Jesper Dramsch, Florian Pinault
• Skills required:
  • Python
  • Machine Learning
  • Git (Basics)

Note: Only nationals or residents from the ECMWF Member States and Co-operating States are eligible to participate (see Terms and Conditions).

---

Challenge description

Machine learning systems have become competitive in the space of global medium-range weather forecasts. These systems use a variety of models, from convolutional to graph neural networks. Diffusion models are a new type of model that has grown in popularity in 2022. These models are generative models that have the potential to work well on physical problems with high fidelity.

We plan to use the WeatherBench benchmark dataset.

The approach should be multi-tiered. The benefit of WeatherBench is the ability to start out on low-resolution data for experimentation and proof of concept. This data lends itself to fast training and model turn-around, but can scale up to higher resolutions to test the stability of models developed before.

Ideas for the implementation:

1) Start out exploring existing code for diffusion models and compare their viability. Specifically:

• Latent Diffusion Models
• Stable Diffusion Models 2) Build the simplest model on the lowest resolution 3) Experiment with appropriate metrics and evaluate how these impact a possible double penalty 4) Iterate on the model at low resolutions to improve the metrics laid out in the WeatherBench benchmark 5) Move on to higher resolutions when a proof of concept has been achieved.

 

[floriankrb]

Great! I look forward to seeing your application.

[Laudrup21]

Hi all, I am really interested and want to work on this project.

[trakasa]

Hi all, I am really interested and want to work on this project.

That's great! We look forward to receiving your application! If you have any concrete questions, please use the remaining time until 12. April to ask them here...

[trakasa]

Hi all, I am really interested and want to work on this project.

That's great! We look forward to receiving your application! If you have any concrete questions, please use the remaining time until 12. April to ask them here...

[trakasa]

Hi all, I am really interested and want to work on this project.

That's great! We look forward to receiving your application! If you have any concrete questions, please use the remaining time until 12. April to ask them here...

[Laudrup21]

I want to know how to write a good proposal?

[jonathanwider]

Hi! Thanks for proposing this cool topic! A friend of mine and I are very interested in participating in the challenge.

I have two questions regarding the challenge:

• Regarding the auto-encoding step in latent/stable diffusion:
  • I understand “stable/latent diffusion” is as a two-step process, consisting of 1) training an Auto-Encoder to reduce the spatial dimension and 2) the Diffusion model that operates in the reduced latent space. The original stable diffusion paper (https://doi.org/10.48550/arXiv.2112.10752) uses a “perceptive” loss function when training the auto-encoder: they claim, by doing this they try to only keep structure that is important for the visual perception of the image while disregarding things that the human eye can’t perceive (camera noise, …). However, To me it isn’t clear, whether in case of weather data, we can make a similar assumption and only focus on “human vision” - after all, what we want in the end is a good prediction and not a “good looking weather map”. Thus, I’m wondering whether this prevents us from using pre-trained auto-encoders for our weather data. And whether we can adapt their loss function when training a custom auto-encoder for weather data, which would be the alternative. What is your opinion on this?
• Regarding evaluation metrics:
  • In our opinion, one advantage of using diffusion models for weather predictions is that the “reversing the diffusion” allows an inherently stochastic prediction of weather data. Thus, one could (hopefully) produce an ensemble of future weather states conditioned on the current state. Thus, we think it might be a good idea, to have an aspect of ensemble forecast in the evaluation of the method. Of course it would be ideal, if we could compare the ML ensemble to a numerical weather prediction (NWP) ensemble… but weatherBench only includes a single timeline reanalysis product, if we saw it correctly. Do you think evaluating ensemble aspects is an important point? And do you think it would be possible to get access to corresponding NWP ensemble predictions? I’ve seen some papers evaluating this aspect (e.g. Jonathan Weyn et al., 2021, but for different timescales)

[JesperDramsch]

Hi @jonathanwider,

those are some excellent questions. Thanks for putting those out here.

To be honest, these would be questions I would like to be explored during the course of this project.

I would probably start with pre-trained models to explore how well they transfer and build the capability around applying these models and possibly fine-tuning. Then if that was achieved, move forward to training on more appropriate data sets and loss functions. But this is also a discussion we can have between the mentors and the people doing the actual work to meet both curiosities.

I agree that this would be an interesting target, but it's also much more ambitious than the proposed WeatherBench project. I went with WB first to make this project accessible and to showcase the direction this should go. If the deterministic experiments are successful and promising, it would definitely be interesting to look at ensemble solutions because the diffusion models, as you point out, would lend themselves to this application quite well. However, I don't want this to be the primary goal, to account for the different availabilities and capabilities of mentees in this challenge. As for the data, this would, of course have to be discussed regarding data agreements etc.

[melioristic]

@JesperDramsch Noted. We will adapt our proposal accordingly.