Cancer caller hangs waiting on a futex

[lordkev]

Describe the bug

Cancer caller hangs in a reproducible manner. The caller is started with 24 threads. Slowly the number of active threads appears to dwindle as I can see the CPU usage go down. Eventually only one thread appears to be running as CPU usage is down to ~100% compared to ~2400% at the beginning. After some hours the one remaining thread appears to no longer be active and when connecting to the process via strace it's waiting on a futex.

Version

$ octopus --version
octopus version 0.7.4 (ed012a6e)
Target: x86_64 Linux 5.4.0-73-generic
SIMD extension: AVX2
Compiler: GNU 11.2.0
Boost: 1_76

Command Command line to install octopus:

$ git clone https://github.com/luntergroup/octopus.git
$ octopus/scripts/install.py --dependencies --forests

Command line to run octopus:

/bio/tools/octopus/bin/octopus-0.7.4 -R /bio/ref/hs37d5.fa -I tumor_chr19.bam normal_chr19.bam -o octopus-somatic-4hapmax.chr19.vcf.gz --normal-sample normal --bamout oct_0.7.4-bams-chr19 --forest /bio/tools/octopus/resources/forests/germline.v0.7.4.forest --somatic-forest /bio/tools/octopus/resources/forests/somatic.v0.7.4.forest --max-somatic-haplotypes 4 --normal-contamination-risk LOW --regions 19 --debug octopus_0.7.4_chr19.log --threads 24

Additional context The tumor bam in this case is a synthetic mixture of data that I created by subsetting reads from bams of two germline samples. However all other chromosomes aside from chr19 complete without issue. I believe this might be related to the report by @jbedo at the end of #150 as they mention very similar behavior of waiting on a futex.

• hs37d5
• Can provide bams and debug log if needed.

[dancooke]

Hi, thanks for the bug report. Please can you provide the output of stdlog?

[lordkev]

Sure, may be a bit as I'll have to run it again. I didn't notice much of interest that looked any different than the rest of the run.

[lordkev]

Hi Dan,

Here is the log file. By this point there was no CPU activity and it appeared to have just deadlocked again.

stdout.log

[dancooke]

Thanks - you're hitting an error/bug:

[2021-08-31 19:14:40] <EROR> Encountered a problem whilst calling 19:23213199-23954187(HaplotypeTree::prune_unique called with matching Haplotype not in tree)

There's also a lot of haplotype skipping which suggests more general problems. Could you can provide the BAMs?

[lordkev]

Ah, sorry I totally missed that - was looking at the bottom of the log. I just emailed you a GDrive link to the BAMs and further explanation.

[dancooke]

Thanks for the BAMs - I'm working on replicating the bug. Could you give a bit more information on how these reads were generated? I'm seeing lot of spliced reads in both the normal and tumour samples that appear like deletions (most 168bp). Here's an IGV pileup showing this, where I've also realigned reads from the normal to called haplotypes:

This is the cause of the many skipped regions and probably why the bug is being triggered.

[lordkev]

Hmm, interesting. The normal sample is the NA12892 platinum genome and it was mapped using BWA-MEM. The only thing a bit out of the norm that I can think of is that read merging was enabled during adapter trimming, though only ~4% of reads were merged.

[jbedo]

Late to this thread but I am also running into this. I'm working on the devel branch atm and consistently have threads getting stuck, and seeing error messages in my logs:

[2022-01-20 22:45:25] <EROR> Encountered a problem whilst calling chr5:1341945-1516474()