Open jperales opened 3 years ago
This appears to be an issue with the way the config.yaml
is written. The cluster
specification is a command line invocation rather than the name of file. If you change cluster from
cluster: "pbs-submit.py --depend \"{dependencies}\""
to
cluster: "pbs-submit.py"
You should find that it now submits jobs (you might also have to make pbs-submit.py
executable (chmod +x ~/.config/snakemake/myprofile/pbs-submit.py
)
Once I've figured how to add dependencies back in to the script invocation I'll update with another comment
Hi @willprice ,
I hope you could also help me to understand how to use this profile on cluster.
I am not sure where I should put the parameters for cluster config.
In my profile/config.yaml
, I tried to add something like cluster: "pbs-submit.py -q tiny"
and chmod +x pbs-submit.py
When I ran snakemake --profile fangz_proflie -rp
Then I got the error of /bin/sh: pbs-submit.py: command not found
Below is the job script printed by snakemake:
Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "fastp_qc", "local": false, "input": ["/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz", "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz"], "output": ["qc/PPMI51971_R1.fastq.gz", "qc/PPMI51971_R2.fastq.gz", "qc/PPMI51971_fastp.json"], "wildcards": {"sample": "PPMI51971"}, "params": {"qualified_quality_phred": 15, "unqualified_percent_limit": 40, "nova_seq": "-g"}, "log": [], "threads": 10, "resources": {"mem": "1gb", "walltime": "04:00:00", "node": 1}, "jobid": 10, "cluster": {}}
cd /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call && \
PATH='/home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin':$PATH /home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin/python3.9 \
-m snakemake qc/PPMI51971_fastp.json --snakefile /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/Snakefile \
--force -j --keep-target-files --keep-remote --max-inventory-time 0 \
--wait-for-files /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.tybds2yj /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/conda/d71cd897 --latency-wait 60 \
--attempt 1 --force-use-threads --scheduler ilp \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
-p --allowed-rules fastp_qc --nocolor --notemp --no-hooks --nolock \
--mode 2 --use-conda --use-singularity && touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.tybds2yj/10.jobfinished || (touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.tybds2yj/10.jobfailed; exit 1)
/bin/sh: pbs-submit.py: command not found
Error submitting jobscript (exit code 127):
I think it did not recognize the cluster configs in profile/config.yaml
(ex: -q tiny
).
Did I add it at the wrong place?
Thanks in advance. Best, Zih-Hua
Hi Zih-Hua,
I'm not the author, but have tried using these scripts.
Try changing your cluster line to cluster: "pbs-submit.py"
If that doesn't work tell me
pbs-submit.py
is locatedI
Hi @willprice,
Thanks a lot for your reply. I understand you're not the author, but I would appreciate that you could share the experience with me.
So I changed the cluster line in profile/config.yaml
to cluster: "pbs-submit.py"
.
Then I got the error qsub: submit error (No default queue specified MSG=requested queue not found)
.
Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "fastp_qc", "local": false, "input": ["/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz", "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz"], "output": ["qc/PPMI51971_R1.fastq.gz", "qc/PPMI51971_R2.fastq.gz", "qc/PPMI51971_fastp.json"], "wildcards": {"sample": "PPMI51971"}, "params": {"qualified_quality_phred": 15, "unqualified_percent_limit": 40, "nova_seq": "-g"}, "log": ["logs/fastp_PPMI51971.log"], "threads": 10, "resources": {"mem": "1gb", "walltime": "04:00:00", "node": 1}, "jobid": 10, "cluster": {}}
cd /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call && \
PATH='/home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin':$PATH /home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin/python3.9 \
-m snakemake qc/PPMI51971_fastp.json --snakefile /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/Snakefile \
--force -j --keep-target-files --keep-remote --max-inventory-time 0 \
--wait-for-files /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.o3d0fpyf /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/conda/d71cd897 --latency-wait 60 \
--attempt 1 --force-use-threads --scheduler ilp \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
-p --allowed-rules fastp_qc --nocolor --notemp --no-hooks --nolock \
--mode 2 --use-conda --use-singularity && touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.o3d0fpyf/10.jobfinished || (touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.o3d0fpyf/10.jobfailed; exit 1)
qsub: submit error (No default queue specified MSG=requested queue not found)
Traceback (most recent call last):
File "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fangz_profile/pbs-submit.py", line 155, in <module>
raise e
File "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fangz_profile/pbs-submit.py", line 153, in <module>
res = subprocess.run(cmd, check=True, shell=True, stdout=subprocess.PIPE)
File "/home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/lib/python3.9/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'qsub -l "nodes=1:ppn=10,mem=1gb,walltime=04:00:00" /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.o3d0fpyf/snakejob.fastp_qc.10.sh' returned non-zero exit status 191.
Error submitting jobscript (exit code 1):
I then created a cluster_config.yml
in the profile
directory.
__default__:
-j: oe
-q: short
I ran snakemake --profile /path/to/fangz_profile --cluster-config /path/to/fangz_profile/cluster_config.yml -rp
.
Then the parameters were correctly parsed "cluster": {"-j": "oe", "-q": "short"}
, but I still got the error of qsub: submit error (No default queue specified MSG=requested queue not found)
. cmd did not print my cluster config.
Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "fastp_qc", "local": false, "input": ["/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz", "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz"], "output": ["qc/PPMI51971_R1.fastq.gz", "qc/PPMI51971_R2.fastq.gz", "qc/PPMI51971_fastp.json"], "wildcards": {"sample": "PPMI51971"}, "params": {"qualified_quality_phred": 15, "unqualified_percent_limit": 40, "nova_seq": "-g"}, "log": ["logs/fastp_PPMI51971.log"], "threads": 10, "resources": {"mem": "1gb", "walltime": "04:00:00", "node": 1}, "jobid": 10, "cluster": {"-j": "oe", "-q": "short"}}
cd /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call && \
PATH='/home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin':$PATH /home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/bin/python3.9 \
-m snakemake qc/PPMI51971_fastp.json --snakefile /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/Snakefile \
--force -j --keep-target-files --keep-remote --max-inventory-time 0 \
--wait-for-files /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.31x25uer /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R1.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fastq/PPMI51971_R2.fastq.gz /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/conda/d71cd897 --latency-wait 60 \
--attempt 1 --force-use-threads --scheduler ilp \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
-p --allowed-rules fastp_qc --nocolor --notemp --no-hooks --nolock \
--mode 2 --use-conda --use-singularity && touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.31x25uer/10.jobfinished || (touch /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.31x25uer/10.jobfailed; exit 1)
qsub: submit error (No default queue specified MSG=requested queue not found)
Traceback (most recent call last):
File "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fangz_profile/pbs-submit.py", line 155, in <module>
raise e
File "/beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/fangz_profile/pbs-submit.py", line 153, in <module>
res = subprocess.run(cmd, check=True, shell=True, stdout=subprocess.PIPE)
File "/home/tu/tu_tu/tu_knkph01/miniconda3/envs/snakemake/lib/python3.9/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'qsub -l "nodes=1:ppn=10,mem=1gb,walltime=04:00:00" /beegfs/work/workspace/ws/tu_knkph01-short_read-0/SR_var_call/.snakemake/tmp.31x25uer/snakejob.fastp_qc.10.sh' returned non-zero exit status 191.
Error submitting jobscript (exit code 1):
Below is my snakemake directory structure where I put my profile folder (fangz_profile):
.
├── config.yaml
├── envs
│ └── short_read_mapping.yml
├── fangz_profile
│ ├── cluster_config.yml
│ ├── config.yaml
│ ├── pbs-jobscript.sh
│ ├── pbs-status.py
│ └── pbs-submit.py
├── fastq
│ ├── PPMI54144_R1.fastq.gz
│ ├── PPMI54144_R2.fastq.gz
│ ├── PPMI56954_R1.fastq.gz
│ └── PPMI56954_R2.fastq.gz
├── hg38
│ ├── 1000G_omni2.5.hg38.vcf.gz
│ └──1000G_omni2.5.hg38.vcf.gz.tbi
├── rules
│ ├── align.smk
│ ├── haplotype_caller.smk
│ └── qc_split.smk
└── Snakefile
Appreciate your insights!
The first error is a good one, it tells us that snakemake can invoke pbs-submit.py
correctly.
I'm not familiar with profiles in snakemake, but looking at the slurm profile I see they set some options in
https://github.com/Snakemake-Profiles/slurm/blob/master/%7B%7Bcookiecutter.profile_name%7D%7D/config.yaml
but it's not immediately obvious how their job submit scripts use them. Perhaps you can delve into those to see how it works and apply your findings to your scenario.
Sorry not to be of more help.
I just was lazy and set my defaults within the pbs-submit.py
script which got the job done!
I understand. Maybe I should just type all my qsub parameters coupled with the --cluster-config
without using profile like in the old version of snakemake.
Unfortunately, I have a lot of parameters that need to be decided on the per-rule basis.
Thanks anyway! Appreciate your time!
No worries, in that case I'd try and reverse engineer one of the better maintained configs in this organisation (e.g. the slurm one) to see how they get the parameters from the profile. For overrides from your rule I think you can just use snakemake.utils.read_job_properties
Popping in to say that the solution outlined by @willprice worked for me when I also made the pbs-status.py
executable. I am also trying to figure out how to best provide rule-specific cluster parameters.
@zihhuafang I found this repository that I think has a much more intuitive profile
implementation.
I basically copied it and modified the qsub
command as necessary and added a cluster.yaml
file for my pipeline.
# ~/.snakemake/pbs-torque/config.yaml
cluster-config: "config/cluster.yaml"
cluster: "qsub -N {cluster.name} -l nodes={cluster.nodes} -l mem={cluster.mem} -l walltime={cluster.walltime} -j oe -o {cluster.out}"
jobs: 10
latency-wait: 60
verbose: true
# ~/projects/myproject/config/cluster.yaml
__default__:
name: default
walltime: 4:00:00
mem: 8gb
nodes: 1:ppn=8
out: pbs_out/
Then I run my command as snakemake <target> --profile pbs-torque
.
@gibsramen Thanks for the link. I actually found this repository as well and am currently using it as the intermediate solution for me. This was how I used to run my snakemake (the old version). I will probably find time to make pbs profile works as the slurm one as I do prefer to specify resources in the rules.
Hi, I am trying to use this profile in our HPC facility (PBS-torque schedule system). However it crashes when I use
snakemake --config ~/.config/snakemake/pbs-torque/config.yaml
, with the following error:I don't understand the problem. I have checked that shell and python are installed in our HPC. I have also tried to edit the first line of
pbs-submit.py
andpbs-jobscript.sh
to point out to alternative different paths for the interpretersshell
andpython3
; but it still does not work.I have checked that the rule for the job run well in an interactive job session in a node of the cluster. So it is just the profile what it is not working. Thanks in advanced for your help! Javier
The log and jobscript for one of the jobs is the following: