Prepending extra scheduler commands in scripts

[adegomme]

Hi, I want to setup a code (or a computer) with slurm, and set the project/partition parameters once and for all in the setup phase.

For this I set some prepend_text when asked during the code setup

SBATCH --account=xxx

SBATCH --partition=yyy

This works well. But if in my project I setup environment variables through the metadata, they get inserted in the resulting submission script before these lines, which are then ignored by slurm.

The script then looks like

SBATCH --cpus-per-task=4

SBATCH --time=00:30:00

ENVIRONMENT VARIABLES BEGIN

export OMP_NUM_THREADS='1'

ENVIRONMENT VARIABLES END

SBATCH --account=xxx

SBATCH --partition=yyy

The only option I see then is to give the queue_name and account options to the scheduler through the calcjob, hence having to write them explicitely in my notebooks everytime I want to create a calculator, which is not very practical as they are not meant to change.

Would it be possible to insert the variables after the prepend_text in the submission scripts ? Or maybe adding an "extra scheduler commands" in the scheduler plugins ? I could subclass the slurm one, but I would prefer doing otherwise.

[zhubonan]

This is indeed a limitation for SLURM (for other like SGE it is OK to put scheduler commands in Code). I can see that having a Code specific scheduler commands can be useful in some cases, but it is difficult to balance the convenience and generalisation. On the other hand, you can just write a helper function to set up the account/queue, since one already needs to put resources and wallclock limit into option, one extra call is not that much.

[ltalirz]

Hi @adegomme , does the custom_scheduler_commands field do the trick?

https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/calculations/usage.html?highlight=environment_variables#options

In any case,

1. I agree that we might want to move the environment variables section after the prepend text for the reasons you explain. Thoughts @giovannipizzi ?
2. As yet another workaround, instead of using the environment_variables, you could also just add export OMP_NUM_THREADS='1' to your prepend_text.

[giovannipizzi]

Agreed with comment 2 of Leo and on the use of the custom_scheduler_commands (that are there exactly for this reason).

I am not sure that swapping the order of prepend text and environment variables is a good solution. I see the prepend text as something run as close as possible to the actual run. Also, swapping might change behaviour for other users (e.g. if one relies on the env vars being already set and manipulates them).

One thing that is probably missing instead (and we could convert this issue to this): allow to define custom_scheduler_commands in the computer configuration, so you can define them once and for all for a computer (now this is not possible IIRC, you can only define the prepend text).