Memory troubles

[lt11]

Dear developers,

I'm running out of memory despite using a bed file. I'm working with a large genome (zebrafish) so that's not a big surprise (although I'm using up to 256 GB of RAM: Picked up _JAVA_OPTIONS: -Xms32g -Xmx256g

Yet, I get this error: Critical exception occurs on region: 2:1-59640629, program will be stopped. Critical exception occurs on region: 3:1-62628489, program will be stopped. java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: Java heap space at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:683) at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658) at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094) at com.astrazeneca.vardict.modes.AbstractMode.pipeline(AbstractMode.java:49) at com.astrazeneca.vardict.modes.SomaticMode.processBothBamsInPipeline(SomaticMode.java:111) at com.astrazeneca.vardict.modes.SomaticMode.access$000(SomaticMode.java:31) at com.astrazeneca.vardict.modes.SomaticMode$SomdictWorker.call(SomaticMode.java:96) at com.astrazeneca.vardict.modes.SomaticMode$SomdictWorker.call(SomaticMode.java:79) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.OutOfMemoryError: Java heap space at htsjdk.samtools.SAMUtils.getAlignmentBlocks(SAMUtils.java:742) at htsjdk.samtools.SAMRecord.getAlignmentBlocks(SAMRecord.java:1789) at htsjdk.samtools.SAMRecord.validateCigar(SAMRecord.java:1807) at htsjdk.samtools.BAMRecord.getCigar(BAMRecord.java:284) at htsjdk.samtools.SAMRecord.isValid(SAMRecord.java:2103) at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:848) at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:834) at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:802) at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.advance(BAMFileReader.java:1079) at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.(BAMFileReader.java:1048) at htsjdk.samtools.BAMFileReader.createIndexIterator(BAMFileReader.java:959) at htsjdk.samtools.BAMFileReader.query(BAMFileReader.java:612) at htsjdk.samtools.SamReader$PrimitiveSamReaderToSamReaderAdapter.query(SamReader.java:533) at htsjdk.samtools.SamReader$PrimitiveSamReaderToSamReaderAdapter.query(SamReader.java:538) at htsjdk.samtools.SamReader$PrimitiveSamReaderToSamReaderAdapter.queryOverlapping(SamReader.java:400) at com.astrazeneca.vardict.data.SamView.(SamView.java:21) at com.astrazeneca.vardict.modules.VariationRealigner.noPassingReads(VariationRealigner.java:2060) at com.astrazeneca.vardict.modules.VariationRealigner.realigndel(VariationRealigner.java:617) at com.astrazeneca.vardict.modules.VariationRealigner.realignIndels(VariationRealigner.java:387) at com.astrazeneca.vardict.modules.VariationRealigner.process(VariationRealigner.java:88) at com.astrazeneca.vardict.modes.AbstractMode$$Lambda$43/0x00007ed9acccf458.apply(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:680) at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658) at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094) at com.astrazeneca.vardict.modes.AbstractMode.pipeline(AbstractMode.java:49) at com.astrazeneca.vardict.modes.SomaticMode.processBothBamsInPipeline(SomaticMode.java:111) at com.astrazeneca.vardict.modes.SomaticMode.access$000(SomaticMode.java:31) at com.astrazeneca.vardict.modes.SomaticMode$SomdictWorker.call(SomaticMode.java:96) at com.astrazeneca.vardict.modes.SomaticMode$SomdictWorker.call(SomaticMode.java:79) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

It is a bit puzzling to me having a "critical exception occurs on region: 2:1-59640629". In fact, this region was split in 10 kb overlapping windows.

Any suggestions?

Thanks in advance.

Best regards,

Lorenzo

[PolinaBevad]

Hi Lorenzo, what is the command line you are using? If you provide BED file, it should show you only the error for the specific region and not the full chromosome. Maybe you can share part of your BED file as well.

[lt11]

Hi Polina,

Thanks for the quick reply. I was running the following lines: vardict -G "${ref_path}" -f "${min_af}" \ -th "${n_threads}" \ -b "${popu_bam}|${ctrl_bam}" \ -N "${popu_samp[ind_i]}" --nosv \ --fisher \ -c 1 -S 2 -E 3 "${path_callable_bed}" | \ var2vcf_paired.pl -N "${popu_samp[ind_i]}|${ctrl_samp[ind_i]}" \ -f "${min_af}" \

"${tmp_vcf}"

I have attached a copy of the bed file specified in the variable "${path_callable_bed}".

Best,

L

call-regions-wins.txt

[lt11]

Maybe it is related to the last (short) window of each chromosome? Yet, this would not explain why I get an error for the full chromosome.

L

[lt11]

Ups, I think I may have found a typo. Let me check it before you spend time on this. I'll get back to you as soon as possible.

L

[PolinaBevad]

Yes, just a note looking in traceback: the error originated from realignment algorithm, if you disable it using -k option should go through. There seems to be a bug in determining the end of the possible region to realign deletion, but hard to say why it occurs without looking at the data. We have a limit of the length for the region to realign as the longest read length from original BAM including soft clips. Is everything fine with the original data? Are there any long reads?

[lt11]

Hi, indeed there was a typo (which I noticed following your comment). I was not reading the correct bed file. I've just run the corrected script. I'll keep you posted. Thanks a lot!

L

[lt11]

The original data is a set of short-reads mapped with a standard (i.e. super-tested) pipeline based on BWA.

L

[lt11]

Hi again,

Indeed it was the typo that prevented the correct bed file (with the windows) to be loaded. Everything works smoothly now.

Again, thanks a lot for the reply. It really helped me finding the typo.

I'll close the issue.

Best,

L