BOILERPLATE_HEADER file in process_fastq.bash gets prematurely deleted by first job to finish

[gturenchalk]

When running the pipeline in my environment (Red Hat Enterprise Linux Server release 6.5 (Santiago) with 24 CPUs available 49402264k memory), I am running into trouble with the BOILERPLATE_HEADER cleanup step of process_fastq.bash:

process_fastq.bash 160 #begin cleanup 161 rm -f ${BOILERPLATE_HEADER} Here are some debug logging excerpts to illustrate the issue:

• echo 'REFERENCE = /remote/rigdata/vol17/gturenchalk2/HLA_hackathon/pipeline/references/all_chr.fa'
• BOILERPLATE_HEADER=/home2/gturenchalk2/tmp/boilerplate_header.bcl
• [[ ! -f /home2/gturenchalk2/tmp/boilerplate_header.bcl ]]
• cat (the above shows the first call that results in a "cat" to instantiate the header file)

---//---

• rm -f /home2/gturenchalk2/tmp/boilerplate_header.bcl
• echo 'completed processing file ./294904' (the above shows the cleanup step being called)

---//---

• samtools mpileup -RB -C 0 -Q 0 -f /remote/rigdata/vol17/gturenchalk2/HLA_hackathon/pipeline/references/all_chr.fa.gz ./tutorialTest6B/final/454Reads.MID8.fastq.contigs.bwa.sorted.bam
• cat /home2/gturenchalk2/tmp/boilerplate_header.bcl -
• bcftools view -O z -o ./tutorialTest6B/final/454Reads.MID8.fastq.vcf.gz cat: /home2/gturenchalk2/tmp/boilerplate_header.bcl: No such file or directory (the above shows that the removal step is being called prior to the completion of all jobs so that when an attempt to include a "cat" of the header file as part of a string of piped commands fails)

My guess is that the script is not waiting for the jobs to finish and the cleanup step is being deleted by the first job to finish. Remaining jobs still needing to use the file will have already passed the header creation test point and end up failing to find the header file they are expecting.

[janders3-nmdp]

closing issue, as code above appears to resolve it. please reopen or file a new issue if i am incorrect.