search

hammerlab / biokepi

Bioinformatics Ketrew Pipelines
Apache License 2.0
27 stars 4 forks source link

Record tool random failures #193

Open smondet opened 8 years ago

smondet commented 8 years ago

Some tools can fail if weird ways and succeed when restarted. Where should the knowledge go?

Here is a bwa-mem:

[M::main_mem] read 2400000 sequences (240000000 bp)...
*** Error in `bwa': free(): invalid next size (fast): 0x00007f06e0001570 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7cfe1)[0x7f0843788fe1]
bwa[0x43953a]
bwa[0x43976c]
bwa[0x43f423]
/lib64/libpthread.so.0(+0x7dc5)[0x7f0843ad4dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f084380221d]
======= Memory map: ========
00400000-0044b000 r-xp 00000000 00:2a 3665962                            /....../my-playground/bwa/bwa
0064b000-0064c000 r--p 0004b000 00:2a 3665962                            /.../my-playground/bwa/bwa
0064c000-0064d000 rw-p 0004c000 00:2a 3665962                            /./my-playground/bwa/bwa
00b8c000-2c38b000 rw-p 00000000 00:00 0                                  [heap]
7f06d8000000-7f06d807a000 rw-p 00000000 00:00 0 
7f06d807a000-7f06dc000000 ---p 00000000 00:00 0 
7f06dc000000-7f06dc07c000 rw-p 00000000 00:00 0 
7f06dc07c000-7f06e0000000 ---p 00000000 00:00 0 
7f06e0000000-7f06e004c000 rw-p 00000000 00:00 0 
7f06e004c000-7f06e4000000 ---p 00000000 00:00 0 
7f06e5cf5000-7f06e64f6000 rw-p 00000000 00:00 0 
7f06e64f6000-7f06e64f7000 ---p 00000000 00:00 0 
7f06e64f7000-7f06e6cf7000 rw-p 00000000 00:00 0                          [stack:41739]
7f06e6cf7000-7f06e6cf8000 ---p 00000000 00:00 0 
7f06e6cf8000-7f06e74f8000 rw-p 00000000 00:00 0                          [stack:41738]
7f06e74f8000-7f06e750d000 r-xp 00000000 08:03 7602195                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f06e750d000-7f06e770c000 ---p 00015000 08:03 7602195                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f06e770c000-7f06e770d000 r--p 00014000 08:03 7602195                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f06e770d000-7f06e770e000 rw-p 00015000 08:03 7602195                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f06e770e000-7f06e770f000 ---p 00000000 00:00 0 
7f06e770f000-7f06e7f0f000 rw-p 00000000 00:00 0                          [stack:41737]
7f06e7f0f000-7f06f3f10000 rw-p 00000000 00:00 0 
7f06f3f58000-7f06f3f59000 ---p 00000000 00:00 0 
7f06f3f59000-7f06f4759000 rw-p 00000000 00:00 0                          [stack:41736]
7f06f4759000-7f06f475a000 ---p 00000000 00:00 0 
7f06f475a000-7f06f4f5a000 rw-p 00000000 00:00 0                          [stack:41735]
7f06f4f5a000-7f06f4f5b000 ---p 00000000 00:00 0 
7f06f4f5b000-7f06f575b000 rw-p 00000000 00:00 0                          [stack:41734]
7f06f575b000-7f06f575c000 ---p 00000000 00:00 0 
7f06f575c000-7f06f5f5c000 rw-p 00000000 00:00 0                          [stack:41733]
7f06f5f5c000-7f06f5f5d000 ---p 00000000 00:00 0 
7f06f5f5d000-7f06f675d000 rw-p 00000000 00:00 0                          [stack:41732]
7f06f675d000-7f06f675e000 ---p 00000000 00:00 0 
7f06f675e000-7f06f6f5e000 rw-p 00000000 00:00 0                          [stack:41731]
7f06f6f5e000-7f06f6f5f000 ---p 00000000 00:00 0 
7f06f6f5f000-7f06f775f000 rw-p 00000000 00:00 0                          [stack:41730]
7f06f775f000-7f06f7760000 ---p 00000000 00:00 0 
7f06f7760000-7f06f7f60000 rw-p 00000000 00:00 0                          [stack:41729]
7f06f7f60000-7f06f7f61000 ---p 00000000 00:00 0 
7f06f7f61000-7f06f8761000 rw-p 00000000 00:00 0                          [stack:41728]
7f06f8761000-7f06f8762000 ---p 00000000 00:00 0 
7f06f8762000-7f06f8f62000 rw-p 00000000 00:00 0                          [stack:41727]
7f06f8f62000-7f06f8f63000 ---p 00000000 00:00 0 
7f06f8f63000-7f06f9763000 rw-p 00000000 00:00 0                          [stack:41726]
7f06f9763000-7f06f9764000 ---p 00000000 00:00 0 
7f06f9764000-7f06f9f64000 rw-p 00000000 00:00 0                          [stack:41725]
7f06f9f64000-7f06f9f65000 ---p 00000000 00:00 0 
7f06f9f65000-7f06fa765000 rw-p 00000000 00:00 0                          [stack:41724]
7f06fa765000-7f06fa766000 ---p 00000000 00:00 0 
7f06fa766000-7f06faf66000 rw-p 00000000 00:00 0                          [stack:41723]
7f06faf66000-7f06faf67000 ---p 00000000 00:00 0 
7f06faf67000-7f06fb767000 rw-p 00000000 00:00 0                          [stack:41722]
7f06fb767000-7f06fb768000 ---p 00000000 00:00 0 
7f06fb768000-7f06fbf68000 rw-p 00000000 00:00 0                          [stack:41721]
7f06fbf68000-7f06fbf69000 ---p 00000000 00:00 0 
7f06fbf69000-7f084370c000 rw-p 00000000 00:00 0                          [stack:41720]
7f084370c000-7f08438c2000 r-xp 00000000 08:03 7602906                    /usr/lib64/libc-2.17.so
7f08438c2000-7f0843ac2000 ---p 001b6000 08:03 7602906                    /usr/lib64/libc-2.17.so
7f0843ac2000-7f0843ac6000 r--p 001b6000 08:03 7602906                    /usr/lib64/libc-2.17.so
7f0843ac6000-7f0843ac8000 rw-p 001ba000 08:03 7602906                    /usr/lib64/libc-2.17.so
7f0843ac8000-7f0843acd000 rw-p 00000000 00:00 0 
7f0843acd000-7f0843ae3000 r-xp 00000000 08:03 7602932                    /usr/lib64/libpthread-2.17.so
7f0843ae3000-7f0843ce3000 ---p 00016000 08:03 7602932                    /usr/lib64/libpthread-2.17.so
7f0843ce3000-7f0843ce4000 r--p 00016000 08:03 7602932                    /usr/lib64/libpthread-2.17.so
7f0843ce4000-7f0843ce5000 rw-p 00017000 08:03 7602932                    /usr/lib64/libpthread-2.17.so
7f0843ce5000-7f0843ce9000 rw-p 00000000 00:00 0 
7f0843ce9000-7f0843cfe000 r-xp 00000000 08:03 7603080                    /usr/lib64/libz.so.1.2.7
7f0843cfe000-7f0843efd000 ---p 00015000 08:03 7603080                    /usr/lib64/libz.so.1.2.7
7f0843efd000-7f0843efe000 r--p 00014000 08:03 7603080                    /usr/lib64/libz.so.1.2.7
7f0843efe000-7f0843eff000 rw-p 00015000 08:03 7603080                    /usr/lib64/libz.so.1.2.7
7f0843eff000-7f0844000000 r-xp 00000000 08:03 7602914                    /usr/lib64/libm-2.17.so
7f0844000000-7f08441ff000 ---p 00101000 08:03 7602914                    /usr/lib64/libm-2.17.so
7f08441ff000-7f0844200000 r--p 00100000 08:03 7602914                    /usr/lib64/libm-2.17.so
7f0844200000-7f0844201000 rw-p 00101000 08:03 7602914                    /usr/lib64/libm-2.17.so
7f0844201000-7f0844222000 r-xp 00000000 08:03 7602899                    /usr/lib64/ld-2.17.so
7f084430b000-7f084440f000 rw-p 00000000 00:00 0 
7f084441e000-7f0844422000 rw-p 00000000 00:00 0 
7f0844422000-7f0844423000 r--p 00021000 08:03 7602899                    /usr/lib64/ld-2.17.so
7f0844423000-7f0844424000 rw-p 00022000 08:03 7602899                    /usr/lib64/ld-2.17.so
7f0844424000-7f0844425000 rw-p 00000000 00:00 0 
7ffdd8b31000-7ffdd8b54000 rw-p 00000000 00:00 0                          [stack]
7ffdd8bbe000-7ffdd8bc0000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
smondet commented 8 years ago

Wget, doing FTP to get Strelka:

Logging in as strelka ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /v1-branch/v1.0.14 ... done.
==> SIZE strelka_workflow-1.0.14.tar.gz ... 7002878
==> PASV ... done.    ==> RETR strelka_workflow-1.0.14.tar.gz ... done.
Length: 7002878 (6.7M) (unauthoritative)
*** buffer overflow detected ***: wget terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7fad85a58ac7]
/lib64/libc.so.6(+0x10bc80)[0x7fad85a56c80]
/lib64/libc.so.6(+0x10da37)[0x7fad85a58a37]
wget[0x406a59]
wget[0x406c47]
wget[0x406eac]
wget[0x425edc]
...
smondet commented 8 years ago

GATK indel-realigner failed with:

##### ERROR ------------------------------------------------------------------------------------------
##### ERROR A USER ERROR has occurred (version 3.5-0-g36282e4): 
##### ERROR
##### ERROR This means that one or more arguments or inputs in your command are incorrect.
##### ERROR The error message below tells you what is the problem.
##### ERROR
##### ERROR If the problem is an invalid argument, please check the online documentation guide
##### ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
##### ERROR
##### ERROR Visit our website and forum for extensive documentation and answers to 
##### ERROR commonly asked questions http://www.broadinstitute.org/gatk
##### ERROR
##### ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
##### ERROR
##### ERROR MESSAGE: Invalid command line: Malformed walker argument: Could not find walker with name: RealignerTargetCreator
##### ERROR ------------------------------------------------------------------------------------------

Some chormosomes succeeded and others failed.

Restarting seems to work.

smondet commented 8 years ago

When Seq2HLA runs on non-human data, it looks like this:

# reads processed: 0
# reads with at least one reported alignment: 0 (0.00%)
# reads that failed to align: 0 (0.00%)
No alignments
/usr/lib/R/bin/exec/R: /tmp/_MEIZyDRNr/libtinfo.so.5: no version information available (required by /tmp/_MEIZyDRNr/libreadline.so.6)
Traceback (most recent call last):
  File "<string>", line 1361, in <module>
  File "<string>", line 126, in main
  File "<string>", line 179, in mainClassI
  File "<string>", line 1244, in expression
ValueError: could not convert string to float: /tmp/_MEIZyDRNr/libtinfo.so.5
seq2HLA returned -1
arahuja commented 8 years ago

bwa mem failures:

[M::main_mem] read 1669990 sequences (120000049 bp)...
*** Error in `bwa': free(): invalid next size (fast): 0x00000000190c50a0 ***
ExecScript.sh: line 37: 31490 Broken pipe             samtools bam2fq '/demeter/scratch/ahujaa01/ksinai-demeter/work/normal-75N-tumor-75T-noRNA-hg38/75T-read-name-sorted.bam'
     31491 Segmentation fault      | bwa mem -R '@RG\tID:75T-read-name-sorted.bam\tSM:75T-read-name-sorted\tLB:ga\tPL:Illumina' -t 12 -O 11 -E 4 '/demeter/scratch/ahujaa01/ksinai-demeter/reference-genome/hg38/hg38.fasta' -p -
     31492 Done                    | samtools view -b
     31493 Done                    | samtools sort -@ 12 -T /demeter/scratch/ahujaa01/ksinai-demeter/work/normal-75N-tumor-75T-noRNA-hg38/tumor-75T-75T.bam_bwamem-default-tmp -o '/demeter/scratch/ahujaa01/ksinai-demeter/work/normal-75N-tumor-75T-noRNA-hg38/tumor-75T-75T.bam_bwamem-default.bam'
arahuja commented 8 years ago

Failure in samtools bam2fq | bwa mem

[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 9259)
[M::mem_pestat] mean and std.dev: (1592.73, 1820.02)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 12314)
[M::mem_pestat] skip orientation RR

paired reads have different names: "XYZ_0055:2:1101:1177:29462#07", "XYZ_0055:2:1101:1177:53090#07"

[mem_sam_pe] [mem_sam_pe]
tavinathanson commented 7 years ago

When running samtools index:

[W::bam_hdr_read] EOF marker is absent. The input is probably truncated.
samtools index: "/nfs-pool-2/biokepi/work/results-b37decoy-dna_processing_137/1b75bb82f758c48a9a243a123afc4b54_checkpoint-trials_lung_ipinivo_tumor_bams_AB653591-5_bamAB653591-5-sorted.bam" is corrupted or unsorted

From @armish:

definitely a truncated BAM (failed sorting causes this). I think you only need to remove the intermediate file and you should be good to go (that is what I do when I hit that issue)