search

PapenfussLab / gridss

GRIDSS: the Genomic Rearrangement IDentification Software Suite
Other
250 stars 73 forks source link

Error during assembly #599

Open YiBenqiu opened 2 years ago

YiBenqiu commented 2 years ago

Dear Developers. I am performing the assembly step with 15X WGS data (~400 samples) with a reference genome size of ~2.7G. But java reports an error. Is this prompt for insufficient memory? Here is my script and the intercepted error message. may be I should use less threads like -t 4? Also limited by computing resources, I don't know if it is correct for me to use array to split the asemble steps into 15 separate tasks as well as running 4 at a time?

#!/bin/bash
#SBATCH --job-name=assemble.sbatch
#SBATCH -A inspurgroup
#SBATCH --partition=fatPartiton,computerPartiton
#SBATCH --nodes=1
#SBATCH --cpus-per-task=24
#SBATCH --mem=257G
#SBATCH --array=0-14%4
#SBATCH --output=assemble_%A_%a.out
#SBATCH --error=assemble_%A_%a.err
echo "SLURM_JOB_ID: $SLURM_JOB_ID"
 ......
......
 ......
gridss \
   -r $ref \
   -j /home/qiuyb/bio_soft/gridss_v213/gridss-2.13.2-gridss-jar-with-dependencies.jar \
   -s assemble \
   -a $workdir/assembly.bam \
   -w $workdir \
   -b $blacklist \
   --jobnodes 15 \
   --jobindex $SLURM_ARRAY_TASK_ID \
   -t 8 \
   --jvmheap 256g \
   --otherjvmheap  256g \
   ${sample_bam} 2> $workdir/assembly_${SLURM_ARRAY_TASK_ID}.log

and the error: INFO 2022-07-31 23:39:53 PositionalAssembler Error during assembly of chromosome 5 (39648 reads in graph). Attempting recovery by rebuilding assembly graph. java.lang.RuntimeException: Exception during asynchronous read at htsjdk.samtools.util.AsyncReadTaskRunner.raiseAsynchronousProcessingException(AsyncReadTaskRunner.java:193) at htsjdk.samtools.util.AsyncReadTaskRunner.raiseAsynchronousProcessingException(AsyncReadTaskRunner.java:187) at htsjdk.samtools.util.AsyncReadTaskRunner.nextRecord(AsyncReadTaskRunner.java:160) at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:865) at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:841) at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:804) at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.advance(BAMFileReader.java:1192) at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1182) at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1146) at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:574) at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:553) at au.edu.wehi.idsv.util.BufferedIterator.fillBuffer(BufferedIterator.java:28) at au.edu.wehi.idsv.util.BufferedIterator.next(BufferedIterator.java:38) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:636) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at au.edu.wehi.idsv.DirectedEvidenceIterator.ensureBuffer(DirectedEvidenceIterator.java:35) at au.edu.wehi.idsv.DirectedEvidenceIterator.hasNext(DirectedEvidenceIterator.java:65) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at au.edu.wehi.idsv.util.AutoClosingIterator.hasNext(AutoClosingIterator.java:47) at au.edu.wehi.idsv.util.AutoClosingIterator.hasNext(AutoClosingIterator.java:47) at au.edu.wehi.idsv.util.AutoClosingMergedIterator.tryclose(AutoClosingMergedIterator.java:82) at au.edu.wehi.idsv.util.AutoClosingMergedIterator.next(AutoClosingMergedIterator.java:61) at au.edu.wehi.idsv.util.DensityThrottlingIterator.ensureNext(DensityThrottlingIterator.java:49) at au.edu.wehi.idsv.util.DensityThrottlingIterator.hasNext(DensityThrottlingIterator.java:90) at au.edu.wehi.idsv.DirectedEvidenceDensityThrottlingIterator.hasNext(DirectedEvidenceDensityThrottlingIterator.java:71) at au.edu.wehi.idsv.DirectedEvidenceErrorCorrectingIterator.addToCurrentBucket(DirectedEvidenceErrorCorrectingIterator.java:72) at au.edu.wehi.idsv.DirectedEvidenceErrorCorrectingIterator.fillCurrentBucket(DirectedEvidenceErrorCorrectingIterator.java:64) at au.edu.wehi.idsv.DirectedEvidenceErrorCorrectingIterator.ensureLastBucket(DirectedEvidenceErrorCorrectingIterator.java:57) at au.edu.wehi.idsv.DirectedEvidenceErrorCorrectingIterator.hasNext(DirectedEvidenceErrorCorrectingIterator.java:49) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127) at com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1324) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127) at au.edu.wehi.idsv.debruijn.positional.PositionalAssembler$ReferenceIndexIterator.hasNext(PositionalAssembler.java:384) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127) at au.edu.wehi.idsv.debruijn.positional.SupportNodeIterator.advance(SupportNodeIterator.java:219) at au.edu.wehi.idsv.debruijn.positional.SupportNodeIterator.ensureBuffer(SupportNodeIterator.java:211) at au.edu.wehi.idsv.debruijn.positional.SupportNodeIterator.hasNext(SupportNodeIterator.java:192) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127) at au.edu.wehi.idsv.debruijn.positional.AggregateNodeIterator.process(AggregateNodeIterator.java:72) at au.edu.wehi.idsv.debruijn.positional.AggregateNodeIterator.ensureBuffer(AggregateNodeIterator.java:56) at au.edu.wehi.idsv.debruijn.positional.AggregateNodeIterator.hasNext(AggregateNodeIterator.java:37) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127) at au.edu.wehi.idsv.debruijn.positional.PathNodeIterator.advance(PathNodeIterator.java:64) at au.edu.wehi.idsv.debruijn.positional.PathNodeIterator.next(PathNodeIterator.java:109) at au.edu.wehi.idsv.debruijn.positional.PathNodeIterator.next(PathNodeIterator.java:22) at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1150) at au.edu.wehi.idsv.debruijn.positional.NonReferenceContigAssembler.nextPosition(NonReferenceContigAssembler.java:359) at au.edu.wehi.idsv.debruijn.positional.NonReferenceContigAssembler.advanceUnderlying(NonReferenceContigAssembler.java:420) at au.edu.wehi.idsv.debruijn.positional.NonReferenceContigAssembler.advanceUnderlying(NonReferenceContigAssembler.java:371) at au.edu.wehi.idsv.debruijn.positional.NonReferenceContigAssembler.ensureCalledContig(NonReferenceContigAssembler.java:273) at au.edu.wehi.idsv.debruijn.positional.NonReferenceContigAssembler.hasNext(NonReferenceContigAssembler.java:197) at au.edu.wehi.idsv.debruijn.positional.PositionalAssembler.flushIfRequired(PositionalAssembler.java:77) at au.edu.wehi.idsv.debruijn.positional.PositionalAssembler.ensureAssembler(PositionalAssembler.java:293) at au.edu.wehi.idsv.debruijn.positional.PositionalAssembler.ensureAssembler(PositionalAssembler.java:107) at au.edu.wehi.idsv.debruijn.positional.PositionalAssembler.hasNext(PositionalAssembler.java:62) at au.edu.wehi.idsv.AssemblyEvidenceSource.assembleChunk(AssemblyEvidenceSource.java:246) at au.edu.wehi.idsv.AssemblyEvidenceSource.assembleChunk(AssemblyEvidenceSource.java:199) at au.edu.wehi.idsv.AssemblyEvidenceSource.lambda$assembleBreakends$2(AssemblyEvidenceSource.java:102) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at htsjdk.samtools.util.AsyncReadTaskRunner.processNextBatch(AsyncReadTaskRunner.java:258) at htsjdk.samtools.util.AsyncReadTaskRunner$$Lambda$53/690686166.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) ... 3 more INFO 2022-07-31 23:39:53 PositionalAssembler java.lang.RuntimeException: Exception during asynchronous read at htsjdk.samtools.util.AsyncReadTaskRunner.raiseAsynchronousProcessingException(AsyncReadTaskRunner.java:193) at htsjdk.samtools.util.AsyncReadTaskRunner.raiseAsynchronousProcessingException(AsyncReadTaskRunner.java:187) at htsjdk.samtools.util.AsyncReadTaskRunner.nextRecord(AsyncReadTaskRunner.java:160) at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:865) at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:841) at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:804) at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.advance(BAMFileReader.java:1192) at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1182) at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1146) at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:574) at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:553) at au.edu.wehi.idsv.util.BufferedIterator.fillBuffer(BufferedIterator.java:28) at au.edu.wehi.idsv.util.BufferedIterator.next(BufferedIterator.java:38) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:636) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at au.edu.wehi.idsv.DirectedEvidenceIterator.ensureBuffer(DirectedEvidenceIterator.java:35) at au.edu.wehi.idsv.DirectedEvidenceIterator.hasNext(DirectedEvidenceIterator.java:65) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at au.edu.wehi.idsv.util.AutoClosingIterator.hasNext(AutoClosingIterator.java:47) at au.edu.wehi.idsv.util.AutoClosingIterator.hasNext(AutoClosingIterator.java:47) at au.edu.wehi.idsv.util.AutoClosingMergedIterator.tryclose(AutoClosingMergedIterator.java:82) at au.edu.wehi.idsv.util.AutoClosingMergedIterator.next(AutoClosingMergedIterator.java:61) at au.edu.wehi.idsv.util.DensityThrottlingIterator.ensureNext(DensityThrottlingIterator.java:49) at au.edu.wehi.idsv.util.DensityThrottlingIterator.hasNext(DensityThrottlingIterator.java:90) at au.edu.wehi.idsv.DirectedEvidenceDensityThrottlingIterator.hasNext(DirectedEvidenceDensityThrottlingIterator.java:71) at au.edu.wehi.idsv.DirectedEvidenceErrorCorrectingIterator.addToCurrentBucket(DirectedEvidenceErrorCorrectingIterator.java:72) at au.edu.wehi.idsv.DirectedEvidenceErrorCorrectingIterator.fillCurrentBucket(DirectedEvidenceErrorCorrectingIterator.java:64) at au.edu.wehi.idsv.DirectedEvidenceErrorCorrectingIterator.ensureLastBucket(DirectedEvidenceErrorCorrectingIterator.java:57) at au.edu.wehi.idsv.DirectedEvidenceErrorCorrectingIterator.hasNext(DirectedEvidenceErrorCorrectingIterator.java:49) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127) at com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1324) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127) at au.edu.wehi.idsv.debruijn.positional.PositionalAssembler$ReferenceIndexIterator.hasNext(PositionalAssembler.java:384) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127) at au.edu.wehi.idsv.debruijn.positional.SupportNodeIterator.advance(SupportNodeIterator.java:219) at au.edu.wehi.idsv.debruijn.positional.SupportNodeIterator.ensureBuffer(SupportNodeIterator.java:211) at au.edu.wehi.idsv.debruijn.positional.SupportNodeIterator.hasNext(SupportNodeIterator.java:192) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127) at au.edu.wehi.idsv.debruijn.positional.AggregateNodeIterator.process(AggregateNodeIterator.java:72) at au.edu.wehi.idsv.debruijn.positional.AggregateNodeIterator.ensureBuffer(AggregateNodeIterator.java:56) at au.edu.wehi.idsv.debruijn.positional.AggregateNodeIterator.hasNext(AggregateNodeIterator.java:37) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127) at au.edu.wehi.idsv.debruijn.positional.PathNodeIterator.advance(PathNodeIterator.java:64) at au.edu.wehi.idsv.debruijn.positional.PathNodeIterator.next(PathNodeIterator.java:109) at au.edu.wehi.idsv.debruijn.positional.PathNodeIterator.next(PathNodeIterator.java:22) at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1150) at au.edu.wehi.idsv.debruijn.positional.NonReferenceContigAssembler.nextPosition(NonReferenceContigAssembler.java:359) at au.edu.wehi.idsv.debruijn.positional.NonReferenceContigAssembler.advanceUnderlying(NonReferenceContigAssembler.java:420) at au.edu.wehi.idsv.debruijn.positional.NonReferenceContigAssembler.advanceUnderlying(NonReferenceContigAssembler.java:371) at au.edu.wehi.idsv.debruijn.positional.NonReferenceContigAssembler.ensureCalledContig(NonReferenceContigAssembler.java:273) at au.edu.wehi.idsv.debruijn.positional.NonReferenceContigAssembler.hasNext(NonReferenceContigAssembler.java:197) at au.edu.wehi.idsv.debruijn.positional.PositionalAssembler.flushIfRequired(PositionalAssembler.java:77) at au.edu.wehi.idsv.debruijn.positional.PositionalAssembler.ensureAssembler(PositionalAssembler.java:293) at au.edu.wehi.idsv.debruijn.positional.PositionalAssembler.ensureAssembler(PositionalAssembler.java:107) at au.edu.wehi.idsv.debruijn.positional.PositionalAssembler.hasNext(PositionalAssembler.java:62) at au.edu.wehi.idsv.AssemblyEvidenceSource.assembleChunk(AssemblyEvidenceSource.java:246) at au.edu.wehi.idsv.AssemblyEvidenceSource.assembleChunk(AssemblyEvidenceSource.java:199) at au.edu.wehi.idsv.AssemblyEvidenceSource.lambda$assembleBreakends$2(AssemblyEvidenceSource.java:102) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at htsjdk.samtools.util.AsyncReadTaskRunner.processNextBatch(AsyncReadTaskRunner.java:258) at htsjdk.samtools.util.AsyncReadTaskRunner$$Lambda$53/690686166.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) ... 3 more

INFO 2022-07-31 23:43:13 PositionalAssembler Error during assembly of chromosome 5 (16948 reads in graph). Attempting recovery by rebuilding assembly graph. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:492) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:128) at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1127)

YiBenqiu commented 2 years ago

My idea was to use fewer threads and to split the assembly steps into 50 separate tasks, as well as running 4 tasks at a time, e.g. :

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=257G
#SBATCH --array=0-49%4
#SBATCH --output=assemble_%A_%a.out
#SBATCH --error=assemble_%A_%a.err
 ......
......
 ......
gridss \
   -r $ref \
   -j /home/qiuyb/bio_soft/gridss_v213/gridss-2.13.2-gridss-jar-with-dependencies.jar \
   -s assemble \
   -a $workdir/assembly.bam \
   -w $workdir \
   -b $blacklist \
   --jobnodes 50 \
   --jobindex $SLURM_ARRAY_TASK_ID \
   -t 4 \
   --jvmheap 256g \
   --otherjvmheap  256g \
   ${sample_bam} 2> $workdir/assembly_${SLURM_ARRAY_TASK_ID}.log

Is this operation correct?

d-cameron commented 2 years ago

Is this prompt for insufficient memory?

Yes, it is. The root cause error message is buried in the middle and is this:

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

My idea was to use fewer threads and to split the assembly steps into 50 separate tasks, as well as running 4 tasks at a time, e.g. :

Reducing the number of parallel tasks per process and increasing the available memory is indeed the correct approach.

15X WGS data (~400 samples)

Are you doing joint calling of all 400 samples (i.e. 6000x coverage input to the assembly), or running them separately?

YiBenqiu commented 2 years ago

Are you doing joint calling of all 400 samples (i.e. 6000x coverage input to the assembly), or running them separately?

Yes, I'm doing a joint calling of all 400 samples. According to the instructions ('If you can do joint assembly, do so.') I entered all 400 samples in the -s assemble step, this task is run on the slurm platform, so I used --array=0-49%4 in #sbatch and also set --jobnodes 50 --jobindex $SLURM_ARRAY_TASK_ID in gridss to span 50 nodes to run only 4 tasks at a time.