Description
Currently, the MDTF holds onto all of the preprocessed data in the WORK_DIR, and it only removes the data in each POD's dir. This PR goes in an removes the rest of the .nc files if "save_pp_data" is set to false in the config file. It is done by using the "case_info.yml" file found in each POD dir. This will help conserve space for runs of the MDTF that involve many PODs.
Associated issue #702
How Has This Been Tested?
The framework was ran on multiple GFDL datasets and with various PODs on my workstation.
Checklist:
[x] My branch is up-to-date with the NOAA-GFDL main branch, and all merge conflicts are resolved
[x] The scripts are written in Python 3.12 or above (preferred; required if funded by a CPO grant), NCL, or R
[ ] All of my scripts are in the diagnostics/[POD short name] subdirectory, and include a main_driver script, template html, and settings.jsonc file
[ ] I have made corresponding changes to the documentation in the POD's doc/ subdirectory
[ ] I have requested that the framework developers add packages required by my POD to the python3, NCL, or R environment yaml file if necessary, and my environment builds with conda_env_setup.sh
[ ] I have added any necessary data to input_data/obs_data/[pod short name] and/or input_data/model/[pod short name]
[ ] My code is portable; it uses MDTF environment variables, and does not contain hard-coded file or directory paths
[ ] I have provided the code to generate digested data files from raw data files
[ ] Each digested data file generated by the script contains numerical data (no figures), and is 3 GB or less in size
[ ] I have included copies of the figures generated by the POD in the pull request
[x] The repository contains no extra test scripts or data files
Description Currently, the MDTF holds onto all of the preprocessed data in the WORK_DIR, and it only removes the data in each POD's dir. This PR goes in an removes the rest of the .nc files if "save_pp_data" is set to false in the config file. It is done by using the "case_info.yml" file found in each POD dir. This will help conserve space for runs of the MDTF that involve many PODs.
Associated issue #702
How Has This Been Tested? The framework was ran on multiple GFDL datasets and with various PODs on my workstation.
Checklist:
conda_env_setup.sh