search

broadinstitute / pilon

Pilon is an automated genome assembly improvement and variant detection tool
GNU General Public License v2.0
338 stars 60 forks source link

long runtime #114

Closed dcopetti closed 4 years ago

dcopetti commented 4 years ago

Hello, I am polishing (first iteration on this assembly) a plant genome assembly with this command: java -Xmx480G -jar /home/copettid/anaconda3/share/pilon-1.23-2/pilon-1.23.jar --genome flye_8kb_191004_assembly_p2.fa --frags PE700_to_OF1_p2_s.bam --output flye_8kb_191004_assembly_p3.fa --diploid --threads 44 --changes The assembly is about 3.6 Gb, the bam file is 111 GB, the process is using about 500 GB RAM and all the 30 processors I asked for. What I am concerned about is the runtime: it is now in its 11th day and no output has been written yet. A similar (3.4 Gb, 36 GB bam file) assembly took about two days - I wonder if something is going wrong with the current polishing step. Thanks!

dcopetti commented 4 years ago

Update: the run stopped after 24 days. These are the last lines of the stdout:

fix break: contig_19004:116892-117150 117095 -426 +425 BreakFix
# fix break: contig_19004:117472-118599 0 -0 +0 NoSolution
Finished processing contig_19004:1-130645
Exception in thread "main" htsjdk.samtools.SAMFormatException: Did not inflate expected amount
        at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:147)
        at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:96)
        at htsjdk.samtools.util.BlockCompressedInputStream.inflateBlock(BlockCompressedInputStream.java:550)
        at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:532)
        at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:468)
        at htsjdk.samtools.util.BlockCompressedInputStream.readBlock(BlockCompressedInputStream.java:458)
        at htsjdk.samtools.util.BlockCompressedInputStream.available(BlockCompressedInputStream.java:196)
        at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:331)
        at java.io.DataInputStream.read(DataInputStream.java:149)
        at htsjdk.samtools.util.BinaryCodec.readBytesOrFewer(BinaryCodec.java:418)
        at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:394)
        at htsjdk.samtools.util.BinaryCodec.readByteBuffer(BinaryCodec.java:504)
        at htsjdk.samtools.util.BinaryCodec.readInt(BinaryCodec.java:515)
        at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:187)
        at htsjdk.samtools.BAMFileReader$BAMFileIterator.getNextRecord(BAMFileReader.java:829)
        at htsjdk.samtools.BAMFileReader$BAMFileIndexIterator.getNextRecord(BAMFileReader.java:981)
        at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:803)
        at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:797)
        at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:765)
        at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.advance(BAMFileReader.java:1034)
        at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1024)
        at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:988)
        at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:576)
        at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:548)
        at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:40)
        at scala.collection.Iterator.foreach(Iterator.scala:937)
        at scala.collection.Iterator.foreach$(Iterator.scala:937)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
        at org.broadinstitute.pilon.BamFile.process(BamFile.scala:125)
        at org.broadinstitute.pilon.GenomeRegion.processBam(GenomeRegion.scala:292)
        at org.broadinstitute.pilon.GenomeFile.$anonfun$processRegions$5(GenomeFile.scala:112)
        at org.broadinstitute.pilon.GenomeFile.$anonfun$processRegions$5$adapted(GenomeFile.scala:112)
        at scala.collection.immutable.List.foreach(List.scala:388)
        at org.broadinstitute.pilon.GenomeFile.$anonfun$processRegions$4(GenomeFile.scala:112)
        at org.broadinstitute.pilon.GenomeFile.$anonfun$processRegions$4$adapted(GenomeFile.scala:109)
        at scala.collection.Iterator.foreach(Iterator.scala:937)
        at scala.collection.Iterator.foreach$(Iterator.scala:937)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
        at scala.collection.parallel.ParIterableLike$Foreach.leaf(ParIterableLike.scala:970)
        at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:49)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:63)
        at scala.collection.parallel.Task.tryLeaf(Tasks.scala:52)
        at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:46)
        at scala.collection.parallel.ParIterableLike$Foreach.tryLeaf(ParIterableLike.scala:967)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:149)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:145)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:436)
        at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

no files were written, this is what I used to monitor the run:

(base) bash-4.2$ python process_watcher.py -p 36694 --to dario.copetti@usys.ethz.ch
Watching 1 processes:
PID 36694: java -Xmx480G -jar /home/copettid/anaconda3/share/pilon-1.23-2/pilon-1.23.jar --genome flye_8kb_191004_assembly_p2.fa --frags PE700_to_OF1_p2_s.bam --output flye_8kb_191004_assembly_p3.fa --diploid --threads 44 --changes
 Started: Fri, Jan 10 08:35:09
 Memory (current/peak) - Resident: 523,116,980 / 526,768,236 kB   Virtual: 531,359,132 / 537,717,328 kB
Process stopped
PID 36694: java -Xmx480G -jar /home/copettid/anaconda3/share/pilon-1.23-2/pilon-1.23.jar --genome flye_8kb_191004_assembly_p2.fa --frags PE700_to_OF1_p2_s.bam --output flye_8kb_191004_assembly_p3.fa --diploid --threads 44 --changes
 Started: Fri, Jan 10 08:35:09  Ended: Tue, Feb 04 01:20:05  (duration 24 days, 16:44:55)
 Memory (current/peak) - Resident: 522,990,608 / 526,768,236 kB   Virtual: 530,716,924 / 537,717,328 kB

Any idea why it happened?

SergejN commented 4 years ago

depending on how many contigs you have in the file flye_8kb_191004_assembly_p2.fa, it may be convenient to split the fasta file into multiple chunks and run them in parallel. With this approach, I corrected the axolotl genome (32Gbp in 125K contigs) in a few days. Therefore, 24 days seems a bit too long

dcopetti commented 4 years ago

I agree, for me it worked when splitting assembly and bam file in 20 chunks