search

georgiesamaha / configBuilder-nf

Nextflow config builder scripts for supported HPC infrastructures.
7 stars 1 forks source link

Autodetect which queues are available for detected scheduler #47

Closed georgiesamaha closed 5 months ago

georgiesamaha commented 5 months ago

Automatically detect which queues are available on the HPC system once a job scheduler (pbspro, slurm, sge) is detected.

Closes https://github.com/georgiesamaha/configBuilder-nf/issues/7

georgiesamaha commented 5 months ago

Hello @jfy133, need your assistance to complete this one. Have tested on PBSpro but I don't currently have access to an SGE or SLURM cluster to confirm their queue/partition detection commands work. I've made some assumptions in attempting to detect and parse their queue names.

If you have access to SLURM/SGE could you give it a go and let me know what needs fixing? 🙏

jfy133 commented 5 months ago

I'm very swamped at the moment (in my preparation for the teaching I'm doing in Adelaide), but I'll try to get to it ASAP.

Btw when you have a moment please check Slack about some nf-core related stuff :)

georgiesamaha commented 5 months ago

You're beyond committed to this! Thank you for the effort. Was thinking along the same lines re: try to autoselect a queue based on available resources, but it got too tricky on the system I was testing PBSpro on. Running qstat -Q on this system shows route and execution queues. Looks like this:

Queue              Max   Tot Ena Str   Que   Run   Hld   Wat   Trn   Ext Type
---------------- ----- ----- --- --- ----- ----- ----- ----- ----- ----- ----
normal               0    57 yes yes     0     0     3    54     0     0 Rou*
normal-exec          0  1093 yes yes    10   733   344     0     0     6 Exe*
express              0     0 yes yes     0     0     0     0     0     0 Rou*
express-exec         0    18 yes yes     0     9     9     0     0     0 Exe*

I know this is maybe a pretty unique set up, checked on another system that runs PBSpro and didn't have this problem. Agree it would be very useful function and can discuss when we have more time and add in a future PR.

jfy133 commented 5 months ago

You're beyond committed to this! Thank you for the effort. Was thinking along the same lines re: try to autoselect a queue based on available resources, but it got too tricky on the system I was testing PBSpro on. Running qstat -Q on this system shows route and execution queues. Looks like this:

Queue              Max   Tot Ena Str   Que   Run   Hld   Wat   Trn   Ext Type
---------------- ----- ----- --- --- ----- ----- ----- ----- ----- ----- ----
normal               0    57 yes yes     0     0     3    54     0     0 Rou*
normal-exec          0  1093 yes yes    10   733   344     0     0     6 Exe*
express              0     0 yes yes     0     0     0     0     0     0 Rou*
express-exec         0    18 yes yes     0     9     9     0     0     0 Exe*

I know this is maybe a pretty unique set up, checked on another system that runs PBSpro and didn't have this problem. Agree it would be very useful function and can discuss when we have more time and add in a future PR.

Hahaha, well people not being able to configure their pipelines is becoming a pet peeve of nextflow of mine (not their fault!), so I'm motivated to get this working :tada:

Hm interesting, yeah I think this would the hardest part of the whole project and will require deep investigation into how to (reliable) get the right infromation...