search

CoBrALab / qbatch

The Unlicense
27 stars 13 forks source link

Cannot set multiple default scheduler options via `QBATCH_OPTIONS` #212

Closed mjclarke94 closed 3 years ago

mjclarke94 commented 3 years ago

Unsure if this is something which needs implementing or documenting, but there isn't a clear way to pass in several options via the QBATCH_OPTIONS env variable.

gdevenyi commented 3 years ago

Hi, the documentation states how to provide more options on the command line:

  -o OPTIONS, --options OPTIONS
                        Custom options passed directly to the queuing system
                        (e.g --options "-l vf=8G". This option can be given
                        multiple times (default: [""])

As for the environment variable, its just a string dumped into the job script.

Do you have perhaps an error or a complete description of reproducible problem?

mjclarke94 commented 3 years ago

So by default, the value of options is an empty list. If you set the environment variable, initialises the list with the string as its only entry. The --options flag then appends to this list.

The difference here is that the command line method allows multiple default flags to be set in a platform agnostic manner, whilst the environment variable method allows you to only set one flag (or do some weird workaround with newline characters and manually inserting the #$ for each option after the first.

Having some way to initialise the list with multiple options could be useful:

# Env var setup
export QBATCH_OPTIONS="-P FreeQueue:-A MyBudget"

# Handling
delim = ':'
options = os.environ.get('QBATCH_OPTIONS').split(delim)
gdevenyi commented 3 years ago

According to the qub man page (http://gridscheduler.sourceforge.net/htmlman/htmlman1/qsub.html) arbitrary numbers of options can be specified on a "#$" line.

slurm and pbs both also appear to support arbitrary numbers of options on a single prefixed command line.

I don't see a need for any option here.