search

HCLIMcom / HCLIM2CMOR

Climate model output rewriting of HCLIM climate model data for CORDEX. Based on the COSMO-CLM routines.
GNU General Public License v3.0
0 stars 6 forks source link

fixing sbatch handling to submit all jobs at once (with dependencies) #15

Closed ljoakim closed 6 months ago

ljoakim commented 6 months ago

I've made some changes/fixes, mainly in the master_wrapper.py script, please have a look to see if it makes sense to you:

In commit order:

  1. Two fixes in simulation config.
  2. I've added the simulation name to some intermediate and output folders, and modified directory creation a bit, because I ran simulations in parallel which ended up sharing folders, which resulted in corrupted data.
  3. I changed the add_version_to_outpathsetting in the control_cmor ini file to True, to get a version folder at the end of the output path. It's currently set to latest, but I'm adding it so that the path adheres to the drs.
  4. I've refactored how sbatch jobs are created/submitted in master_wrapper.py. Before when running master_wrapper you had to keep the terminal/python process alive until the cmorization was completely finished. I've changed it so that all jobs are submitted at once, but the cmor and chunk jobs are dependent on the post jobs, so they will not be started until the previous step is finished. Now you can call the master wrapper script and let the jobs run by themselves until they are finished on the slurm machines :)

I also ran black python formatter on master_wrapper.py, which is why some changes are not really related to function, but style.