search

ua-snap / cmip6-utils

Pipelines and utilites for working with CMIP6 data
1 stars 1 forks source link

Fix regridding pipeline and improve granularity #59

Closed kyleredilla closed 3 months ago

kyleredilla commented 4 months ago

This PR updates and fixes the regridding pipeline. It is updated to use the most recent (and hopefully finalized) version of SNAP's CMIP6 ensemble / suite of models, including the MPI-ESM1-2-HR and MRI-ESM2-0 models, and dropping the CESM2-WACCM model. It also hopes to fix some intermittent issues with things getting stuck / hung up in multiprocessing by including some tweaks to make that processing more robust, as well as things like including missing edge years for some models. Additionally, it adds functionality to run the processing scripts on combinations of models, scenarios, and frequencies (monthly or daily), instead of only being split out by variable.

You can test this PR by simply testing the corresponding PR in the prefect repo.

closes #58 (pretty sure this closes this issue, regridding with all variables seems to work fine)

kyleredilla commented 4 months ago

Hey @cstephen, I am going to convert this back to a draft PR, I have been playing around and am still getting issues with things just randomly hanging on tasks that use multiprocessing, which I thought were fixed in this PR. I'll let you know when it's ready for review.