Add optical flow baseline

[jacobbieker]

The baseline is running, but using the HRV channel to compute optical flow (the cloud masks didn't work for computing flow), and then applying the flow to the cloud masks to predict the next 48 timesteps (4 hours) results in a MSE of 0.1541 vs a naive baseline of just using the current frame for all future frame MSE loss of 0.1566. So optical flow gives some improvement, but not a ton. Most of the current networks being trained have MSE of around 0.1-0.07~8 so theoretically are already doing better than optical flow or the naive baseline

[JackKelly]

That's really exciting that the neural networks are doing better than optical flow!

My guess would be that optical flow does really well over short time horizons (30 minutes? 1 hour?). It might be interesting to separately measure error across different forecast horizons (e.g. 30 mins ahead, 1 hr ahead, 2 hrs ahead, and 4 hrs ahead). Some users might be particularly interested in very short horizon forecasts.

[jacobbieker]

Yeah! It saves out the mean error for each timestep, so I do have that information for every 5 minutes once it finishes, its just somewhat slow and running in the background. I'm going to try adding in the error per timestep to the NN outputs soon too, so its easier to compare.

[jacobbieker]

After fixing a bug, the results follow what you expect, where the loss is lower at the beginning and increases steadily as time goes on. On a short horizon, the optical flow, and just using the current image both have losses lower or comparable to the current models I have running

[jacobbieker]

It seems up until about ~1hr in the future, the optical flow and baseline give better results than current models do.

[jacobbieker]

Overall, using the current frame as the output results in a MSE of 0.160, using optical flow gives MSE of 0.157. One of the ConvLSTM models that is training gives a validation MSE of 0.0712 overall, with a MSE of 0.036 for the first future frame, and 0.095 for the last frame, suggesting that the ML models already heavily outperform the baseline models already further in the future, but is comparable or worse for the short term. Still need work on finding possibly better metrics than MSE which might change the comparison, but at least for now it is looking pretty good I think

[JackKelly]

Sounds extremely promising!

[jacobbieker]

Next up will be redoing it with the actual satellite channels, as the binary masks might not be the best way forward #6 The ML models trained to produce the 12 satellite channels tend to have similar loss as for the binary masks, so if that holds for optical flow as well, then the results might be quite similar, which is a good sign for using ML for this!

[jacobbieker]

Optical flow baseline is running now for the 12 sat channels, where the flow is determined for each channel separately. Preliminary results is that the MSE is ~0.9 for optical flow. For models I've been training, their loss tends to be ~0.6-0.3, so the models are potentially much better than optical flow for the actual channels compared to the binary cloud masks!

[jacobbieker]

Plots from the running job. The MSE depends a lot on the specific channel for optical flow. There also seems to be large jumps for certain channels after an hour, which I am not entirely sure why that happens, possibly a bug in the dataloader? Need to investigate that a bit.

[jacobbieker]

The timedelta between all the future timesteps is 5min, like it should be, so not sure for the jump after timestep ~7, but it doesn't seem to be an issue with the dataloader

[jacobbieker]

Things to try: Take the mean of a few pairs of previous images to do the optical flow, and get the optical flow between the last frame and the previous, previous frame, to see if extra movement helps on especially channels that are a bit blurrier and probably harder to compute the flow of

[JackKelly]

Awesome work!

Looking again at the plots, I am really surprised that the MSE for the 'current image' and 'optical flow' are basically identical. If I've understood correct then, if optical flow is doing anything sane, then it really should out-perform 'current image', at least for the first few timesteps of the prediction. But maybe the issue is that I haven't got a good intuitive feel for what MSE of a pair of images means :)

[jacobbieker]

Yeah, I am too. Computing the Optical flow as the average of the optical flow of the last 5 pairs of images gives more of a difference for the MSE.