Enable pass through of extra scheduler arguments

[jwhite242]

Slurm has numerous optional arguments for both sbatch and srun. Rather than trying to enumerate all of them with their own keys in the batch/step blocks, I'd like to propose adding a way to pass through user defined arguments. Consider something like the following examples, using the normal yaml key:val syntax nested inside a new registered key such as 'extra_args'

batch:
    type        : slurm
    host        : quartz
    bank        : baasic
    queue       : pbatch
    gres        : ignore
    reservation : test_reservation
    extra_args:
        begin: "02:30:00"            # delay execution by 2.5 hours
        depends: 238474            # start after job with id 238474
        ... 

study:
    ...
    - name: run-lulesh
      description: Run LULESH.
      run:
          cmd: |
            $(LAUNCHER) $(LULESH)/build/lulesh2.0 -s $(SIZE) -i $(ITERATIONS) -p > $(outfile)
          depends: [make-lulesh]
          nodes: 2
          procs: 27
          exclusive   : True
          walltime: "00:10:00"
          extra_args:
              cpus-per-task: 4            # Run task with 4 threads

So in this case the batch block extra_args would get dumped into the sbatch header while the extra_args in the step would be for the $(LAUNCHER) expansion.

This does put some of the syntax checking on the user side of course, has issues as well. This might be even more useful for schedulers like Flux that are still changing more rapidly, making any baked in syntax checking/registration more fragile.

[FrankD412]

So, I've encountered this when working with Flux also. There are extra options that can be specified. Your suggestion would collect all of those in one place for users who need a level of tweaking for specific schedulers. Just documenting that here as this isn't the first time I'm encountering this use case.

[youngjeffrey]

Hi Jeremy, what specific keys were you thinking of including in the "extra args"? In addition what kind of behavior are you expecting with the inclusion of extra arguments?

[jwhite242]

Well, I was thinking it'd be more of a general purpose thing rather than any specific keys as the list of available/optional keys on these schedulers is absurdly long and can even vary across machines.

[jwhite242]

So, for the behavior, i suppose some of them might need to get pulled into the launcher token, while others probably work fine in batch script headers. Hmm.. That makes it a little more complicated then; maybe there needs to be a separate key for the two types (can possibly have some that can be used in both?).