1) Bugs in Freebayes sometimes cause it to crash randomly mid-chromosome
2) Perf stat doesn't report the correct exit code, so the pipeline thinks it passed without errors.
I can try fixing it by updating Freebayes, every recipe was using the same Freebayes binary. But, this means it would change for "frozen" recipes too.
Here are a lot more details on the two issues:
Issue 1:
The first issue: Freebayes occasionally throws errors and exits with a 134. Here is the error message:
currentSequence matching:
terminate called after throwing an instance of 'std::out_of_range'
what(): basic_string::substr
There is a long discussion about this issue on the Freebayes GitHub repo. It looks like people are still reporting it as a bug. But it's possible that upgrading to a newer version will fix it. https://github.com/ekg/freebayes/issues/6
Issue 2:
Perf stat should always report the exit code of the command it's wrapping, but in this case it does not:
[rrichholt@dback-login1]/scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/oeFiles$ perf stat /home/tgenref/pecan/bin/freebayes/bin/freebayes -f /home/tgenref/pecan/bwa_index/hs37d5_plusRibo_plusOncoViruses_plusERCC.fa -b /scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/KHS5U/FDAV_COLO829m100x0_1_CL_Whole_T1_S5U_L03520/FDAV_COLO829m100x0_1_CL_Whole_T1_S5U_L03520.proj.md.jr.bam -b /scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/KHS5U/FDAV_COLO829m0x100_1_CL_Whole_C1_S5U_L03522/FDAV_COLO829m0x100_1_CL_Whole_C1_S5U_L03522.proj.md.jr.bam -t /home/tgenref/pipeline_v0.4/chrListBED/dir2/Step16.bed --ploidy 2 --min-repeat-entropy 1 > /scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/freebayes/FDAV_COLO829m0x100_1_CL_Whole_C1_S5U_L03522-FDAV_COLO829m100x0_1_CL_Whole_T1_S5U_L03520/FDAV_COLO829m0x100_1_CL_Whole_C1_S5U_L03522-FDAV_COLO829m100x0_1_CL_Whole_T1_S5U_L03520_Step16.freebayes.vcf
Exception: Unable to read reference sequence base past end of current cached sequence.
16:7058722
7058932-7059044
alignment: CCCTCCCTTCCTCCCTCCCTTCTTTCCTCCCTCCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTTCCTCCCTTCTTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCT
currentSequence: GGCGCATCTGTAATTCCAGACACTCAGAAGTCAGAGGCAGCAGAATCGCTTGAACCTAGGAAGCAAAGGTTGCAGTGATCTGAGATCATACCACTGCACTCCAGCCTGGGTGACAGGGCAAGACTTTGTCTCAAAAAAAAAGTAGAATTCAGTGAACAAGAGTAAATAATTCAAGAAATTCAAGGTTGATAGCTATAAAAATGAAATTCCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCACTTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTC
currentSequence matching:
Exception: Unable to read reference sequence base past end of current cached sequence.
16:7058722
7058978-7059071
alignment: CTTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTTCCTTCCTTCCTCCCCCCCTTCCTCCCTCCCTTCTTCCCTCCGTCCCTTCCTCCC
currentSequence: GGCGCATCTGTAATTCCAGACACTCAGAAGTCAGAGGCAGCAGAATCGCTTGAACCTAGGAAGCAAAGGTTGCAGTGATCTGAGATCATACCACTGCACTCCAGCCTGGGTGACAGGGCAAGACTTTGTCTCAAAAAAAAAGTAGAATTCAGTGAACAAGAGTAAATAATTCAAGAAATTCAAGGTTGATAGCTATAAAAATGAAATTCCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCACTTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTC
currentSequence matching:
terminate called after throwing an instance of 'std::out_of_range'
what(): basic_string::substr
/home/tgenref/pecan/bin/freebayes/bin/freebayes: Aborted
Even though Freebayes crashed after only 1/10th of the chromosome was complete, the exit code is reported as 0 by perf stat. I noticed it while porting the pipeline to Slurm because I removed the perf stat wrappers on the commands since Slurm accounting can handle that job now.
1) Bugs in Freebayes sometimes cause it to crash randomly mid-chromosome 2) Perf stat doesn't report the correct exit code, so the pipeline thinks it passed without errors.
I can try fixing it by updating Freebayes, every recipe was using the same Freebayes binary. But, this means it would change for "frozen" recipes too.
Here are a lot more details on the two issues:
Issue 1:
The first issue: Freebayes occasionally throws errors and exits with a 134. Here is the error message:
currentSequence matching: terminate called after throwing an instance of 'std::out_of_range' what(): basic_string::substr
There is a long discussion about this issue on the Freebayes GitHub repo. It looks like people are still reporting it as a bug. But it's possible that upgrading to a newer version will fix it. https://github.com/ekg/freebayes/issues/6
Issue 2:
Perf stat should always report the exit code of the command it's wrapping, but in this case it does not:
[rrichholt@dback-login1]/scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/oeFiles$ perf stat /home/tgenref/pecan/bin/freebayes/bin/freebayes -f /home/tgenref/pecan/bwa_index/hs37d5_plusRibo_plusOncoViruses_plusERCC.fa -b /scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/KHS5U/FDAV_COLO829m100x0_1_CL_Whole_T1_S5U_L03520/FDAV_COLO829m100x0_1_CL_Whole_T1_S5U_L03520.proj.md.jr.bam -b /scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/KHS5U/FDAV_COLO829m0x100_1_CL_Whole_C1_S5U_L03522/FDAV_COLO829m0x100_1_CL_Whole_C1_S5U_L03522.proj.md.jr.bam -t /home/tgenref/pipeline_v0.4/chrListBED/dir2/Step16.bed --ploidy 2 --min-repeat-entropy 1 > /scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/freebayes/FDAV_COLO829m0x100_1_CL_Whole_C1_S5U_L03522-FDAV_COLO829m100x0_1_CL_Whole_T1_S5U_L03520/FDAV_COLO829m0x100_1_CL_Whole_C1_S5U_L03522-FDAV_COLO829m100x0_1_CL_Whole_T1_S5U_L03520_Step16.freebayes.vcf Exception: Unable to read reference sequence base past end of current cached sequence. 16:7058722 7058932-7059044 alignment: CCCTCCCTTCCTCCCTCCCTTCTTTCCTCCCTCCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTTCCTCCCTTCTTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCT currentSequence: GGCGCATCTGTAATTCCAGACACTCAGAAGTCAGAGGCAGCAGAATCGCTTGAACCTAGGAAGCAAAGGTTGCAGTGATCTGAGATCATACCACTGCACTCCAGCCTGGGTGACAGGGCAAGACTTTGTCTCAAAAAAAAAGTAGAATTCAGTGAACAAGAGTAAATAATTCAAGAAATTCAAGGTTGATAGCTATAAAAATGAAATTCCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCACTTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTC currentSequence matching: Exception: Unable to read reference sequence base past end of current cached sequence. 16:7058722 7058978-7059071 alignment: CTTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTTCCTTCCTTCCTCCCCCCCTTCCTCCCTCCCTTCTTCCCTCCGTCCCTTCCTCCC currentSequence: GGCGCATCTGTAATTCCAGACACTCAGAAGTCAGAGGCAGCAGAATCGCTTGAACCTAGGAAGCAAAGGTTGCAGTGATCTGAGATCATACCACTGCACTCCAGCCTGGGTGACAGGGCAAGACTTTGTCTCAAAAAAAAAGTAGAATTCAGTGAACAAGAGTAAATAATTCAAGAAATTCAAGGTTGATAGCTATAAAAATGAAATTCCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCACTTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTCCCTCCCTCCCTTCCTCCCTCCCTTCCTCCCTCCCTCCCTTCCTCCCTCCCTC currentSequence matching: terminate called after throwing an instance of 'std::out_of_range' what(): basic_string::substr /home/tgenref/pecan/bin/freebayes/bin/freebayes: Aborted
Performance counter stats for '/home/tgenref/pecan/bin/freebayes/bin/freebayes -f /home/tgenref/pecan/bwa_index/hs37d5_plusRibo_plusOncoViruses_plusERCC.fa -b /scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/KHS5U/FDAV_COLO829m100x0_1_CL_Whole_T1_S5U_L03520/FDAV_COLO829m100x0_1_CL_Whole_T1_S5U_L03520.proj.md.jr.bam -b /scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/KHS5U/FDAV_COLO829m0x100_1_CL_Whole_C1_S5U_L03522/FDAV_COLO829m0x100_1_CL_Whole_C1_S5U_L03522.proj.md.jr.bam -t /home/tgenref/pipeline_v0.4/chrListBED/dir2/Step16.bed --ploidy 2 --min-repeat-entropy 1':
103,031,568,295 instructions # 1.06 insn per cycle 24,818,887,921 branches # 482.271 M/sec 461,906,920 branch-misses # 1.86% of all branches
[rrichholt@dback-login1]/scratch/rrichholt/jetstream_projects/smallCOLO829_Somatic_ps201709131621/oeFiles$ echo $? 0
Even though Freebayes crashed after only 1/10th of the chromosome was complete, the exit code is reported as 0 by perf stat. I noticed it while porting the pipeline to Slurm because I removed the perf stat wrappers on the commands since Slurm accounting can handle that job now.