dessn / Pippin

Pipeline for photometric SN analysis
MIT License
9 stars 10 forks source link

pleaser add global specifier for BATCH_INFO in pippin input file #14

Closed RickKessler closed 3 years ago

RickKessler commented 3 years ago

Would be really useful to specify GLOBAL: BATCH_INFO: sbatch $PIPPIN_BASE/SBATCH_Cori.TEMPLATE 10

and then remove all of the other BATCH_INFO keys, or replace them with NCORE: 20 to change number cores but keep same SBATCH file. To make this easier to implement, I will modify submit_batch_jobs to accept a new arg, --ncore which overrides NCORE from input file.

Samreay commented 3 years ago

Scaffolding this out - we should define in the cfg.yml a default BATCH_JOB template that points to one of Ricks template files. We can then pass this to all the BATCH_INFO, and use the NCORE argument added to allow for easy changing of the number of jobs without resubmitting anything.

We can then use the BATCH_INFO file in #15 (for SNIRF, NN, others) instead of having our own batch keys

RickKessler commented 3 years ago

--ncore argument ready; see https://github.com/RickKessler/SNANA/issues/300

Samreay commented 3 years ago

So heres how I'd break this task down:

  1. Load in the sbatch templates Rick already has set up - have one SBATCH_CPU and SBATCH_GPU in the cfg.yml, which we read into memory.
  2. We dont want to load the sbatch template file over and over in each task. Have the Manager load them once at the start, fail if they cant be found, and then pass them (either CPU or GPU) to each task like we pass force_refresh and force_ignore.
  3. Create a general function in Task that takes number of cores, GPU or not, memory and the command line args, and use that to build an sbatch command which gets returned.
  4. Update SNN, SNIRF, NN, and Analyse to use this instead of constructing their own. Start with Analyse as it will be the fastest one, and you can have an analyse task with just a DATAPREP and SIM and LCFIT stage if you add MASK_LCFIT so that it makes the data/sim comparison histograms.
OmegaLambda1998 commented 3 years ago

This has been added