Add Conditional Diffusion Model

[jacobbieker]

Detailed Description

Diffusion models seem to be quite useful for a lot of image generation and high detail and much easier training than for GANs as generative models, e.g. StableDiffusion. This seems like we should be able to make an easier to train version of DGMR, or other generative models for forecasting future weather or PV forecasting, as there have been some audio or other 1D diffusion models as well.

Context

Diffusion models have impressive results and should be easier to train than GANs that we have been trying.

Possible Implementation

[jacobbieker]

Video prediction ones: https://github.com/voletiv/mcvd-pytorch https://video-diffusion.github.io/ https://github.com/buggyyang/RVD

[jacobbieker]

https://github.com/newbeeer/poisson_flow

[JackKelly]

This is a super-cool idea, @jacobbieker!

I'd love to chat about this some time!

I've been blown away by recent progress in diffusion models. The temporal coherence in videos produced by Imagen Video is just stunning, and bodes well for the ability to predict weather.

The "video diffusion" paper (Ho et al. 2022) talks a little bit about using their model for video prediction. In section 3.1 they talk about a "reconstruction-guided sampling... based on the model’s reconstruction of the conditioning data". And in section 4.2 they show state-of-the-art results of predicting 11 frames of Kinetics-600 given the first 5 frames. (Relevant to us: They compare very favorably against DVD-GAN, which, if I remember correctly, is the basis for DGMR). I haven't yet fully understood all the maths in their paper but I get the impression that the core idea is probably pretty simple.

Hopefully diffusion models would also give us a principled way to generate an ensemble of predictions (by changing the input noise, similar to your idea of changing the noise input to DGMR).

I also wonder if we can combine your work on the graph weather model with diffusion models... i.e. use a GNN as the denoising model... although I really don't know enough about either GNNs or diffusion models to know if that's a sane idea or not!

[jacobbieker]

Yeah, dvd GAN is the basis of dgmr, so I think we could make a nice improvement over it. For ensemble forecasts, I think just changing the seed or running it multiple times would work too. I think there are some GNN diffusion models, ones that are used for finding proteins? But still trying to figure that out. But yeah, could be quite a good way to combine both approaches. The main issue seems to be that these models generation is somewhat slow compared to other approaches, but there seems to be pretty rapid progress on that front too.

[JackKelly]

Yeah, I'm not too worried about the processing speed at inference time, as long as we're generating a dense prediction (so we don't have to go forwards through the model a million times for a million PV systems!).

BTW, I've been finding the maths a bit hard-going. This video really helped me get closer to understanding the maths. Although I expect you're better at understanding the maths, given your background!

[tcapelle]

for a very simple (Image, timeseries)_t sequence, I was mostly thinking about prediction frames condioned to the timeseries values, or vice-versa.

[jacobbieker]

From a poster at the ECMWF conference that used NCSN++ based model for precipitation forecasting, seems like that type of model is available here: https://github.com/yang-song/score_sde_pytorch which might be a good starting point too

[tcapelle]

I have been playing with this model recently, and I am pretty convinced that it's capable of generating future cloud movement: https://github.com/google-research/frame-interpolation/issues/70 It's pretty simple, using BiFPN and computing motion vectors at different resolutions. The code is super simple (but in TF).

[jacobbieker]

Oh cool! Looks interesting

[jacobbieker]

Related one using diffusion model for atmospheric layers: https://github.com/rybchuk/latent-diffusion-3d-atmopheric-boundary-layer

[tcapelle]

started doing DDPM with clouds: https://wandb.ai/capecape/ddpm_clouds/runs/xq6vlibx?workspace=user-capecape

[tcapelle]

Another cool stuff I discovered, is that the VAE used on the Stable diffusion pipeline is a pretty good encoder of satellite imagery: Letting you drastically reduce the input size:

[Venka97]

Hi @tcapelle, which dataset are you using for training the diffusion model on clouds? This is something that I am also interested in

[tcapelle]

I have access to private data, but It's not a lot. I would love to know if there are public datasets with lots of images.

[Venka97]

Maybe this: https://vision.eng.au.dk/cloudcast-dataset/