search

ua-snap / cmip6-utils

Pipelines and utilites for working with CMIP6 data
1 stars 1 forks source link

Update CMIP6 transfers pipeline for subdaily WRF variables #22

Closed kyleredilla closed 9 months ago

kyleredilla commented 9 months ago

These changes adapt the transfers pipeline to include data that will be suitable for downscaling with WRF. Additionally, there are some other enhancements made to help with clarity as I learned more about the CMIP6 data (via exploring MetaGrid for suitable WRF data) and finalized our model selection.

To review:

  1. follow the README in transfers/ to make sure you can run all of the steps there prior to the final transfers step. For each of the steps that generate an output file tracked in Git, pay attention to whether the file is modified or not. If it is, stop and submit your review there before continuing on with using it moving forward with the transfers pipeline. If that does happen, it would likely just be do to new / updated data upstream, which is probably unlikely.
  2. If the files are not modified while re-running the pipeline, you can try running the transfers step. Try a transfer for a single frequency / variable, and for all data (i.e. just python transfers.py)

Pay attention to what's missing as far as explaining the transfer process in the README, notebooks, and code comments. Like, "What does this notebook do?", "does this comment still apply?" etc. This pipeline is currently a large part of our documentation around this, so I want to make sure it is coherent as possible! There could probably be some quick things added or changed throughout to improve clarity.

Also, no need to review files that are tracked in Git but are not code!

Joshdpaul commented 9 months ago

After reviewing the batch files that had been modified, @kyleredilla and I traced the issue to the removal of the "CESM2" model from the final ensemble. Previous commits (made after @BobTorgerson 's successful testing) had updated the llnl_manifest.csv and llnl_manifest_wrf.csv, but had not yet regenerated the batch files. My testing run removed the "CESM2" files from the batch files, and so these were committed to version control with 45fb326.

Approved!