Open hannahnesser opened 1 week ago
Hi Hannah,
Giving the IMI flexibility to run on different schedulers is great and will help our work at ECCC. We also use PBS Pro but made the changes manually, so this would speed up the process of pulling new IMI updates. Thanks!
Hi Sabour,
Great! If you find that something doesn't work, please feel free to fix it. I haven't done a ton of rigorous testing on this because I don't have much of the methane data downloaded (since I'm applying this to CO2).
Name: Hannah Nesser Institution: JPL
Describe the update
This is a set of changes that:
As of now, this is up to date with the dev branch. The changes include the following (though I may have forgotten something).
PBS/SBATCH flexibility:
UseSlurm
[bool] withSchedulerType
[str].submit_job $SchedulerType SaveOutput JobArguments
whereSaveOutput
is a boolean that determines whether or not the output is appended to imi_output.log. It then callssubmit_slurm_job
orsubmit_pbs_job
as appropriate. Note:submit_pbs_job
currently assumes the use of Pleiades and therefore includes a hardcoded "model" keyword that is required by the system.convert_sbatch_to_pbs
to common.sh that is run in the case thatSchedulerType = "PBS"
. This function searches for all files that haveSBATCH
in their header and then iteratively replaces theSBATCH
commands with thePBS
equivalent. This means that any new SBATCH options added to a header should match the formatting in this function or that the sed statement should be expanded to include the new options.Some changes that were needed to make these changes work:
SBATCH -N 1
uses and instead put the node request intosubmit_job
since PBS requires that the number of nodes be assigned only once and that it occur along with CPU requests, etc. in the initial job request.Conda/pip flexibility:
Methane/CO2 flexibility: