Spatially reproject data in data conversion script

[JackKelly]

Detailed Description

The EUMETSAT data is in geostationary projection. We probably want it in transverse Mercator.

Related

• See the way SatIp used to do it (in git history)
• The notebook I wrote in 2019 to convert .nat files to NetCDF and re-project
• Notebook which reprojects using pyresample
• Notebook which reprojects using rasterio
• pyresample can be told to use multiple CPU cores

[jacobbieker]

SatFlow also did it here: https://github.com/openclimatefix/satflow/blob/ad5d6f6c419994835977cb75322f86b171d6a341/satflow/data/utils/utils.py#L48-L96

[JackKelly]

On second thoughts, maybe we shouldn't spatially reproject the satellite imagery?!?

My thinking is:

• The Perceiver doesn't need different modalities to be aligned. It's fine for data sources to be in different projections (as long as each pixel is correctly encoded with its position).
• Reprojection introduces artifacts.
• Reprojection takes a lot of compute time. Reprojecting multiple years of satellite data might take so long that the data won't be ready in time for our WP1 report in December! And, at inference time, reprojection adds another data processing step, and increases latency.

The downsides I can think of are:

• We wouldn't be able to use a CNN front-end which has receives both satellite data and NWPs as different channels of the same image (because the NWPs and satellite imagery wouldn't be spatially aligned). But we could use separate CNNs for each modaility: one for the satellite imagery; and another for NWPs.
• Ideally, we'd empircally test how reprojection affects forecasting skill. My guess is that reprojection probably hurts because it introduces artifacts. But that is just a guess!
• We wouldn't be allowed to publicly share our non-reprojected satellite data (becuase, to share the data, we need to transform it in some non-trivial way). But that's fine... Separtely, we can reproject the satellite data and release it in January (after our WP1 deadline!)

What do you guys think, @jacobbieker and @peterdudfield

[peterdudfield]

seperate CNNs is fine - lob/master/predict_pv_yield/models/conv3d/conv3d_sat_nwp.png

Does the reporjection helpe with visualisation? Or maybe that reprojection could be done for visualization

[jacobbieker]

Yeah, I can see that not reprojecting it could be better, especially if we end up using other derived EUMETSAT data, like cloud masks or atmospheric instability, keeping everything in the original format makes it a lot easier and yeah, removes artifacts. I'd still want to see if reprojection does do worse or not, but yeah, that could wait. It would help with visualization, but yeah, we can just reproject for that

[JackKelly]

Excellent, sounds like a plan! I won't bother reprojecting for now, and we can see how we get on...