search

ENCODE-DCC / chip-seq-pipeline2

ENCODE ChIP-seq pipeline
MIT License
234 stars 123 forks source link

missing slurm qos parameter #300

Open giardine opened 8 months ago

giardine commented 8 months ago

First I am able to run the pipeline in the open queue, so I know that everything is installed properly. In case it makes a difference, this is a conda install, with Slurm. But I would like to be able to run this under an allocation to avoid long queues. In our system we need to not only define the account and partition but also the qos. This fails with the ENCODE pipelines. In the slurm*.out files I see that the qos is not being used. In default.conf: slurm-partition=burst slurm-qos=burst3x

In slurm*.out: 2023-10-13 12:37:15,399|caper.hpc|INFO| Running shell command: sbatch -t 48:00:00 --mem 4G -p burst -A xxxx --export=ALL -J CAPER_62 /storage/home/bmg137/miaq8ut7.sh sbatch: error: Batch job submission failed: Invalid qos specification

Belinda

leepc12 commented 7 months ago

There is no such slurm-qos parameter in Caper. But you can still customize the leader job submission command line in Caper's conf file (~/.caper/default.conf) here. Also please remove slurm-account if it's not required for your cluster.

slurm-partition=burst
#slurm-account=xxxxxx <------ please remove this if not needed

...

slurm-leader-job-resource-param=-t 48:00:00 --mem 4G   DEFINE_ANY_EXTRA_QOS_PARAMETER_HERE
giardine commented 7 months ago

Thanks. It would be helpful to me if this could be added as an option. I may have jobs running in different accounts and partitions at the same time. They could also get stuck in the queue and then I would not be sure which value it would run with if I am changing the configuration file between submissions. Or even what value any jobs spawned later would get.

This isn't purely theoretical I was actually trying to do this recently. The account is required with the qos parameter.

Belinda

On Mon, 6 Nov 2023, Jin wook Lee wrote:

There is no such slurm-qos parameter in Caper. But you can still customize the leader job submission command line in Caper's conf file (~/.caper/default.conf) here. Also please remove slurm-account if it's not required for your cluster.

slurm-partition=burst
#slurm-account=xxxxxx <------ please remove this if not needed

...

slurm-leader-job-resource-param=-t 48:00:00 --mem 4G   DEFINE_ANY_EXTRA_QOS_PARAMETER_HERE

-- Reply to this email directly or view it on GitHub: https://github.com/ENCODE-DCC/chip-seq-pipeline2/issues/300#issuecomment-1796422044 You are receiving this because you authored the thread.

Message ID: @.***>