Closed dcopetti closed 4 years ago
Update: the run stopped after 24 days. These are the last lines of the stdout:
fix break: contig_19004:116892-117150 117095 -426 +425 BreakFix
# fix break: contig_19004:117472-118599 0 -0 +0 NoSolution
Finished processing contig_19004:1-130645
Exception in thread "main" htsjdk.samtools.SAMFormatException: Did not inflate expected amount
at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:147)
at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:96)
at htsjdk.samtools.util.BlockCompressedInputStream.inflateBlock(BlockCompressedInputStream.java:550)
at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:532)
at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:468)
at htsjdk.samtools.util.BlockCompressedInputStream.readBlock(BlockCompressedInputStream.java:458)
at htsjdk.samtools.util.BlockCompressedInputStream.available(BlockCompressedInputStream.java:196)
at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:331)
at java.io.DataInputStream.read(DataInputStream.java:149)
at htsjdk.samtools.util.BinaryCodec.readBytesOrFewer(BinaryCodec.java:418)
at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:394)
at htsjdk.samtools.util.BinaryCodec.readByteBuffer(BinaryCodec.java:504)
at htsjdk.samtools.util.BinaryCodec.readInt(BinaryCodec.java:515)
at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:187)
at htsjdk.samtools.BAMFileReader$BAMFileIterator.getNextRecord(BAMFileReader.java:829)
at htsjdk.samtools.BAMFileReader$BAMFileIndexIterator.getNextRecord(BAMFileReader.java:981)
at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:803)
at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:797)
at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:765)
at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.advance(BAMFileReader.java:1034)
at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1024)
at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:988)
at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:576)
at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:548)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:40)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at org.broadinstitute.pilon.BamFile.process(BamFile.scala:125)
at org.broadinstitute.pilon.GenomeRegion.processBam(GenomeRegion.scala:292)
at org.broadinstitute.pilon.GenomeFile.$anonfun$processRegions$5(GenomeFile.scala:112)
at org.broadinstitute.pilon.GenomeFile.$anonfun$processRegions$5$adapted(GenomeFile.scala:112)
at scala.collection.immutable.List.foreach(List.scala:388)
at org.broadinstitute.pilon.GenomeFile.$anonfun$processRegions$4(GenomeFile.scala:112)
at org.broadinstitute.pilon.GenomeFile.$anonfun$processRegions$4$adapted(GenomeFile.scala:109)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.parallel.ParIterableLike$Foreach.leaf(ParIterableLike.scala:970)
at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:49)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:63)
at scala.collection.parallel.Task.tryLeaf(Tasks.scala:52)
at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:46)
at scala.collection.parallel.ParIterableLike$Foreach.tryLeaf(ParIterableLike.scala:967)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:149)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:145)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:436)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
no files were written, this is what I used to monitor the run:
(base) bash-4.2$ python process_watcher.py -p 36694 --to dario.copetti@usys.ethz.ch
Watching 1 processes:
PID 36694: java -Xmx480G -jar /home/copettid/anaconda3/share/pilon-1.23-2/pilon-1.23.jar --genome flye_8kb_191004_assembly_p2.fa --frags PE700_to_OF1_p2_s.bam --output flye_8kb_191004_assembly_p3.fa --diploid --threads 44 --changes
Started: Fri, Jan 10 08:35:09
Memory (current/peak) - Resident: 523,116,980 / 526,768,236 kB Virtual: 531,359,132 / 537,717,328 kB
Process stopped
PID 36694: java -Xmx480G -jar /home/copettid/anaconda3/share/pilon-1.23-2/pilon-1.23.jar --genome flye_8kb_191004_assembly_p2.fa --frags PE700_to_OF1_p2_s.bam --output flye_8kb_191004_assembly_p3.fa --diploid --threads 44 --changes
Started: Fri, Jan 10 08:35:09 Ended: Tue, Feb 04 01:20:05 (duration 24 days, 16:44:55)
Memory (current/peak) - Resident: 522,990,608 / 526,768,236 kB Virtual: 530,716,924 / 537,717,328 kB
Any idea why it happened?
depending on how many contigs you have in the file flye_8kb_191004_assembly_p2.fa
, it may be convenient to split the fasta file into multiple chunks and run them in parallel. With this approach, I corrected the axolotl genome (32Gbp in 125K contigs) in a few days. Therefore, 24 days seems a bit too long
I agree, for me it worked when splitting assembly and bam file in 20 chunks
Hello, I am polishing (first iteration on this assembly) a plant genome assembly with this command:
java -Xmx480G -jar /home/copettid/anaconda3/share/pilon-1.23-2/pilon-1.23.jar --genome flye_8kb_191004_assembly_p2.fa --frags PE700_to_OF1_p2_s.bam --output flye_8kb_191004_assembly_p3.fa --diploid --threads 44 --changes
The assembly is about 3.6 Gb, the bam file is 111 GB, the process is using about 500 GB RAM and all the 30 processors I asked for. What I am concerned about is the runtime: it is now in its 11th day and no output has been written yet. A similar (3.4 Gb, 36 GB bam file) assembly took about two days - I wonder if something is going wrong with the current polishing step. Thanks!