search

Snakemake-Profiles / lsf

Snakemake profile for running jobs on an LSF cluster
MIT License
36 stars 22 forks source link

Order of bsub non-standard arguments from lsf.yaml file #36

Closed NBMueller closed 3 years ago

NBMueller commented 3 years ago

https://github.com/Snakemake-Profiles/lsf/blob/c88fcb9fc60ce74596fbf8f516fef31574eef5de/%7B%7Bcookiecutter.profile_name%7D%7D/lsf_config.py#L33

Hi, I would suggest changing the order of rule-specific parameters to first rule_params and then default_params. bsub takes the first argument and ignores any other occurrences, hence the default is always taken. Thanks for the work and best, Nico

mbhall88 commented 3 years ago

Hi @NBMueller. It is actually the other way around, it uses the last option and disregards all others. This is the behaviour I also observe on my system with testing of multiple different options

NBMueller commented 3 years ago

Thanks for the quick reply! Which LSF version is your hpc using? Ours is

>>> lsid
IBM Spectrum LSF Standard 10.1.0.7, Dec 18 2018
Copyright International Business Machines Corp. 1992, 2016.

If I run the following:

bsub -W 0:01 -W 0:02 -J "test" "ps -al"

the job requests 1 min runtime, not 2 mins. Maybe the behavior changes with different LSF versions?

mbhall88 commented 3 years ago

I am using IBM Spectrum LSF Standard 10.1.0.6, May 25 2018. I tested your example and see the same behaviour as you.

It is frustrating because if I do

bsub -q q1 -q q2 -J "test" "sleep 10"

the job gets sent to queue q2. I can't really find much about ordering in the docs...

Given the inconsistencies between options, the best solution seems to be to try and deduplicate all options with the rule-specific options taking priority.

I will work on getting a fix out early next week

mbhall88 commented 3 years ago

@NBMueller let me know if v0.1.0 (i.e. https://github.com/Snakemake-Profiles/lsf/commit/57302c69c9eff456b5a5d727a7ac30b2b684845d) still gives you issues