Automate the processing of gridded data to CSV via shap files

informatics-lab / covid19-examples-and-docs

Documentation and examples for the Met Office Informatics Lab COVID-19 response platform and data.

1 stars 3 forks source link

Automate the processing of gridded data to CSV via shap files #9

Open tam203 opened 4 years ago

tam203 commented 4 years ago

Weekly we update the gridded data.

At the moment this gridded data is uploaded to the cloud blob. Then @kaedonkers runs a notebook (lots of times with different inputs and then uses azcopy to upload the result to the blob.

Can we improve this process?

If we do it Met Office side at the same time we extract the data we could use SPICE and might get performance gains in terms of CPU and disk speed.

@bsherratt

tam203 commented 4 years ago

[ ] @kaedonkers looking at the xaray region mask example from Aaron Spring
[ ] @DPeterK looking at the Iris code we currently have to understand performance issues.