Incorporate other derived EUMETSAT data

[jacobbieker]

Atmospheric Motion Vectors for example could be useful to include where winds seem to go? Still need to look into it more, but there are a few other rapid scan outputs that could be promising. The motion vectors are averaged over 3 5 minute scans, so are only once every 15 minutes though, but still could be useful.

[jacobbieker]

Also has an instability product for the atmosphere at a per pixel level for Europe, so including that could potentially lead to better understanding of where the clouds start to form or disappear in, and could be very helpful for the model, I think

[JackKelly]

I might have misunderstood, but I think @tjvandal's 2019 paper "Optical Flow for Intermediate Frame Interpolation of Multispectral Geostationary Satellite Data" demonstrates that an ML-based optical flow model does a better job of inferring wind motion vectors from satellite imagery than conventional motion vector algorithms :)

(But, even if I've understood correctly, I definitely don't want to stop you trying out the EUMETSAT motion vector product(s)!)

[jacobbieker]

Ah okay, I'll check that out!

[tjvandal]

Work on atmospheric motion vectors (AMVs) and 3D winds is still in progress and motivated by this interpolation study. The interpolation uses flow vectors and visibility maps, and hence the flow vectors are not perfect, though there is certainly a positive correlation. Much more work needs to be done to translate these flows into a physical measurement.

3D winds and AMVs are currently the largest source of uncertainty in numerical weather prediction and data assimilation systems (ie. initializing PDEs are important). I'm working with colleagues from GMAO/NOAA to build models for both flows and cloud height (AMVs need both). So far, results definitely seem to outperform traditional approaches, NOAA/EUMETSAT use similar techniques. It's quite a tricky problem though, there are sparse observations of ground truth winds, unsupervised approaches have not worked yet for me, and cloud heights need to be estimated.

[JackKelly]

Awesome, thanks loads for the details, @tjvandal!

[jacobbieker]

Update: Just finally got the data from eumetsat for nearly all the derived products for RSS, so now can start seeing if including them helps. Some are more temporally sparse, and are only released every 15, 20 or 30 minutes.

[jacobbieker]

Satpy does not currently load the Regional or Atmospheric motion vector BUFR products... https://github.com/pytroll/satpy/issues/1768 hoping that can be fixed soon.

[jacobbieker]

Finally got regional motion vector products for about 6 months of data