Simulated Optical Flow using real clouds

openclimatefix / satflow

Satellite Optical Flow with machine learning models

https://satflow.readthedocs.io/en/stable/

MIT License

61 stars 10 forks source link

Simulated Optical Flow using real clouds #88

Open jacobbieker opened 3 years ago

jacobbieker commented 3 years ago

As mention in #85 one pre-training idea is to create a flow dataset to pre-train on using clouds. We would need simulated flow, and would want to have realistic clouds in all spectral channels. Easiest way to do that, I think, would be to take real clouds and then crop/paste them over the base ground image and move them to create it. So, to do this, there needs to be

[ ] Get clean base image of ground for all timesteps
[ ] Get set of real clouds that we can use for copy/paste, has to be available for all timesteps so its still the correct pixel values as the day goes on
[ ] Create augmentation pipeline to generate images on fly and optical flow

jacobbieker commented 3 years ago

To help, should also add losses for optical flow, like end point error (e2e), one example is here: https://github.com/NVIDIA/flownet2-pytorch/blob/master/losses.py

jacobbieker commented 3 years ago

Easiest way to obtain the clean base image could be just take the average of all of the days for a given place. Another option would be to use the binary cloud mask, and only use the pixel in the average if it is "cloud free". While it could still have some influence from sub-pixel clouds, it'd probably be close enough.

jacobbieker commented 3 years ago

https://arxiv.org/pdf/1511.05440.pdf%5D Also goes into only using loss on parts of the image above a certain optical flow threshold

JackKelly commented 3 years ago

Some very hand-wavy ideas about subtracting 'cloudy pixels' from 'background' here: https://github.com/openclimatefix/predict_pv_yield/issues/17

JackKelly commented 3 years ago

I love these ideas!

But, before diving into data augmentation, is there strong reason to believe the models are overfitting? (although I remember the Perceiver paper emphasising that the Perceiver likes to overfit more than most deep learning models!)

jacobbieker commented 3 years ago

I don't think the models are overfitting actually, from what I'm seeing they aren't yet at least. So yeah, can definitely leave that for later in case they start to overfit!