search

openclimatefix / satflow

Satellite Optical Flow with machine learning models
https://satflow.readthedocs.io/en/stable/
MIT License
59 stars 11 forks source link

Add Video Prediction models #4

Open jacobbieker opened 3 years ago

jacobbieker commented 3 years ago

Various ones to try include https://github.com/thuml/predrnn-pytorch and ConvLSTMs, and potentially GANs as shown in https://arxiv.org/abs/2104.00954

jacobbieker commented 3 years ago

MetNet's work with axial self-attention did really well with precipitation forecasting of up to 8 hours later, and use axial self-attention, which there is a PyTorch implementation here: https://github.com/lucidrains/axial-attention

Interestingly, MetNet is compared to optical flow methods as well in the paper, and does outperform it. Additionally, one important point is the need for a large spatial area around the target of interest to capture changed moving both into and out of the target area, which it does by having the center 64kmx64km crop, at full resolution, then downsampling the 1024x1024km area around the crop. It seems like trying out this approach, because of the similar type of problem makes the most sense as a first try. Trying out different amounts of historical data would also be interesting. MetNet uses the previous 90min at 15min intervals for prediction, since we have 5min observations, we could see if including different number of frames from the previous 90min, or focusing on more recent data (say previous 45min?) might matter more for the next 4 hours.

jacobbieker commented 3 years ago

The 12 spectral bands from SEVRI are not all the same resolution, with the non-HRV being 3km resolution, and HRV being 1km, so need to figure out best way to resample that or potentially just put the non-HRV values at 9 pixels in the HRV image, easily enough.

JackKelly commented 3 years ago

Yeah, in the Zarr file that I've been using, all 12 SEVIRI channels have been down-sampled to something like 4 km resolution.

We have a bunch of code for spatially re-projecting the raw SEVIRI data, and converting to Zarr, if that's useful.

jacobbieker commented 3 years ago

Yeah, I've been looking at that code and messing around with it currently, just working out the best way to integrate all the data together

JackKelly commented 3 years ago

Cool beans. I think the most recent reprojection code is in the satip project.

I'd be tempted to start simple, and start with just one (or a small number) of satellite channels. (And perhaps ignore the High Res Visible channel to start with. I've heard some people suggest that HRV might be one of the less informative channels when it comes to nowcasting clouds, and estimating irradiance at the Earth's surface! But I could be wrong!) That said, I can totally see the attraction of designing a data pipeline from the ground-up that can handle multiple, diverse datasets; so go for it if you want to start there!

jacobbieker commented 3 years ago

After the chat today, seems like a good one to try first would be UNET, MetNet, then something like DVD-GAN. Ideally, UNET should get fairly decent results. Also, try out deeplabv3 .

jacobbieker commented 3 years ago

According to https://www.climatechange.ai/papers/icml2021/54 GANs show promise for predicting future frames from current ones, the code isn't available, but they seem to use a Recurrent Unet + cGAN in their slide.

jacobbieker commented 3 years ago

Pix2Pix does okay trying to predict the 12 satellite channels for the next half hour from the past 20min of data. I tried just half an hour in the future as it was easier to visualize and see what is happening, but plan on going up to 4 hours in the future soonish. This implementation is simple in that it takes a single input image, where the timeseries of images is just stacked along the channel dimension as input, and outputs a multi-channel output image in the same format, so each 12 channels is a single timestep.

These are the generated images after 9 epochs of 2000*16 input image stacks. generated_image

And this is the ground truth for that same timestep target_image

As can be seen, Pix2Pix does get some of the structure present, but is very grainy looking, and fails on the last 2 channels. It does a bit better with this one:

Generated: generated_2

Target: target_2

jacobbieker commented 3 years ago

Another one is https://github.com/FrontierDevelopmentLab/PyRain which does global rain forecasting, mostly seems to be just ConvLSTM, but could have some good stuff on data preparation, etc. Or this one, which the code doesn't seem to be public, but they use a GAN for thunderstorm prediction http://spaceml.org/repo/project/60002402f5647f00129f7287

jacobbieker commented 3 years ago

https://github.com/CNALeon007/TrajGRU/blob/master/TrajGRU.py Trajectory GRU is also interesting, built specifically for precipitation forecasting, it could improve the tracking of clouds through time.

jacobbieker commented 3 years ago

Perciever IO now is also available for predicting dense predictions: https://github.com/deepmind/deepmind-research/tree/master/perceiver

jacobbieker commented 3 years ago

The perceiver IO is now also implemented in PyTorch here: https://github.com/lucidrains/perceiver-pytorch, https://github.com/openclimatefix/perceiver-pytorch/issues/1

jacobbieker commented 2 years ago

PyTorch Video has some neat built in ones, and it might also be helpful to try out https://github.com/facebookresearch/AVT and https://github.com/facebookresearch/SlowFast