search

harvardinformatics / snpArcher

Snakemake workflow for highly parallel variant calling designed for ease-of-use in non-model organisms.
MIT License
69 stars 32 forks source link

Error in rule dedup #123

Closed weirdo-onlooker closed 10 months ago

weirdo-onlooker commented 1 year ago

@erikenbody When I run Snakemake, one of the steps encountered an error, and both the error message and the server administrator indicated that the tmp cache was full.

How can I modify the tmp directory?

Note that after modifying the cache directory in the config.yaml file, there was no change in the behavior.

tsackton commented 1 year ago

Hi,

Which rule encountered the error? Not every job uses big_tmp and it is possible that if your system tmp directory is on the smaller size, you are running into an issue with a rule that we have not modified to use the big_tmp path in the config.

Can you share the error message and any other logs you have?

Edited to add: based on the title of the issue, I guess the problem is in the dedup rule? Currently that rule does not utilize the big_tmp directory.

weirdo-onlooker commented 1 year ago

I'm sorry, I forgot to copy the error message: Error in rule dedup: jobid: 9 input: results/GCF_030020395.1/bams/preMerge/pangolin/PNG1.bam, results/GCF_030020395.1/bams/preMerge/pangolin/PNG1.bam.bai output: results/GCF_030020395.1/bams/pangolin_final.bam, results/GCF_030020395.1/bams/pangolin_final.bam.bai log: logs/GCF_030020395.1/sambambadedup/pangolin.txt (check log file(s) for error details) conda-env: /sdc1/home/hk/yaozc/snpArcher-main/.snakemake/conda/0068d55d38ca6ffb2c000410df01fcd8 shell: sambamba markdup -t 1 results/GCF_030020395.1/bams/preMerge/pangolin/PNG1.bam results/GCF_030020395.1/bams/pangolin_final.bam 2> logs/GCF_030020395.1/sambamba_dedup/pangolin.txt (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.

The pangolin.txt file has been overwritten, and the original message roughly means "no space left."

How can I set dedup to big_tem?

tsackton commented 1 year ago

I created a branch with these changes, can you test and see if this fixes the issue for you?

You will need to update snpArcher to use the dedup-bigtmp branch, either by: running git pull; git checkout dedup-bigtmp in your snpArcher directory, or by running git clone -b dedup-bigtmp https://github.com/harvardinformatics/snpArcher/ in a new directory.

cademirch commented 1 year ago

Curious if @tsackton's fix worked for you @weirdo-onlooker?

cademirch commented 10 months ago

Closing this - @weirdo-onlooker, please reopen if you still have trouble.