Snakemake missing output files reason while using Slurm

[paulaistvan]

Command:

Error

While running normally without submitting job, snakemake is working fine but if i use slurm to submit job via sbatch, then it give missing output files in reason and then job fails.

Tried putting latency wait variable for snakemake also as per internet results that suggested will wait for output files to generate but no luck, any insights on this?

Have tried reducing mem per cpu, no of nodes, cpus etc but no luck.

This is the snakemake rule, pls note it work perfect if we don't submit job via slurm.

[jiarong]

Hi, this is more of a snakemake Q. The issues I can spot are:

• In params, dir=virsorter2_output2/{sample} ==> dir=f"virsorter2_output2/{wildcards.sample}"
• In output, remove touch

[paulaistvan]

Thank you for you quick answer. I added both of your suggestions, but got some other error.

name 'wildcards' is not defined.

and if i simply do the old one in params section but with f added, like this: dir=f"virsorter_output2/{sample}"

then it says, name 'sample' is not defined.

Can you suggest what could be the reason.

[jiarong]

I recommend removing the params, change the shell command to:

virsorter run -i {input.fasta} -d {config[db]} -w virsorter_output2/{wildcards.sample} -j {threads} --min-length 1500 --use-conda-off all

[paulaistvan]

Have changed as per your suggestions, below is final one. It's still giving same error.

Error:

[jiarong]

Are there files produced in virsorter_output2/sample/?

[paulaistvan]

Yes, only this:

[paulaistvan]

But the folder name is "sample" should be the name of the scaffold right?

[jiarong]

Name should be whatever is under "SampleData2". Any message produced from VS2?

[paulaistvan]

yes name is fine, and by message you mean log folder or on the terminal output that you have already seen above? In the log folder, there is only one empty file.

[jiarong]

Hi, sorry that I have not clue what's the issue. The snakemake syntax looks good to me and the VS2 runs OK without snakemake. . It will require the data and some testing to figure out. But I wont be able to solve this snakemake slurm issue for you here.

[erosix]

I have the same issue. Was a solution found? I see that this happens when the rule outputs are not specified in the shell commands directly. Like if the output is a produced file but in the shell command one only specifies the output folder and not the internal files.

[jgroh]

Experiencing a similar issue here.

[ETAOIK]

Experiencing a similar issue here.