search

hwlim / BisKit-RNA

BisKit for CCHMC HPC users
0 stars 0 forks source link

destination folder missing in alignment #4

Closed hwlim closed 4 months ago

hwlim commented 5 months ago

In alignment stage, there is an error of missing output file like this

Rules claiming more threads will be scaled down. Singularity containers: ignored Job counts: count jobs 1 align_se_miRNA 1

[Wed Mar 27 16:09:35 2024] Job 0: Aligning miRNA... [KO_Rapa]

2891043 reads; of these: 2891043 (100.00%) were unpaired; of these: 2870805 (99.30%) aligned 0 times 10493 (0.36%) aligned exactly 1 time 9745 (0.34%) aligned >1 times 0.70% overall alignment rate [E::hts_open] fail to open file '1.Sample/KO_Rapa/3.miRNA/Align/align.bam' Waiting at most 60 seconds for missing files. MissingOutputException in line 439 of /users/limc8h/Programs/BisKit/Snakemake/rules.smk: Job Missing files after 60 seconds: 1.Sample/KO_Rapa/3.miRNA/Align/align.bam 1.Sample/KO_Rapa/3.miRNA/Align/align.bam.bai This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Job id: 0 completed successfully, but some output files are missing. 0 File "/usr/local/python/3.6.3/lib/python3.6/site-packages/snakemake/executors/init.py", line 581, in handle_job_success File "/usr/local/python/3.6.3/lib/python3.6/site-packages/snakemake/executors/init.py", line 259, in handle_job_success Removing output files of failed job align_se_miRNA since they might be corrupted: 1.Sample/KO_Rapa/3.miRNA/Align/align.log, 1.Sample/KO_Rapa/3.miRNA/Align/unaligned_pre.fq.gz Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message

It appears that bam file output is created by samtools sort command but it's destination folder is not created properly before running the command. Thus samtools sort does not run and make error.

samtools sort /scratch/$(whoami)/{wildcards.sampleName}.rRNA.sam -o {output.bam}

Whever there is an output file in a new folder, the folder MUST be created using something like this before hand

mkdir -p ${desDir}

Please update the whole source code considering this.

Yet, interestingly, some folders created without a problem, don't know why. Also, some script or program automatically create output folder on the fly, but samtools doesn't seem to do so.

limc8h@bmiclusterp2:/data/nakamura-lab/0.Analysis.Lim/20220414-RNA_BSeq/BisKit$ samtools sort /scratch/limc8h/KO_Rapa.miRNA.sam -o tmp/test.bam [E::hts_open_format] Failed to open file "tmp/test.bam" : No such file or directory samtools sort: failed to create "tmp/test.bam": No such file or directory

cahn20 commented 4 months ago

This is fixed in pull request #6 (already merged).