search

rrwick / Polypolish

a short-read polishing tool for long-read assemblies
GNU General Public License v3.0
146 stars 10 forks source link

Core dump...... #16

Closed tnn111 closed 10 months ago

tnn111 commented 1 year ago

Hi, I'm getting a core dump. The tail of the log looks like the attached. Key is probably "memory allocation of 655360 bytes failed". Any idea of where this is coming from and how it might be increased?

This is a metagenome and the total assembly is around 4 Gbases. Also, I'm trying to correct using 100 Gbases of Illumina reads...... No problems with SAM although it'd be nice it BAM were supported.

Thanks!

q12_l5_m5_50392 (14,672 bp) q12_l5_m5_50393 (13,213 bp) q12_l5_m5_50394 (8,765 bp) q12_l5_m5_50395 (3,196 bp) q12_l5_m5_50396 (3,427 bp) q12_l5_m5_50397 (34,395 bp) memory allocation of 655360 bytes failed

rrwick commented 1 year ago

I've never tried Polypolish with such a large reference and read set, but based on that error, I think the most likely explanation is that you've run out of memory. How much RAM does your computer have?

You can check maximum memory usage with GNU Time:

/usr/bin/time -v polypolish ...

Then check Maximum resident set size (kbytes) after it finishes/crashes. Note that the exact location of time might depend on your OS.

tnn111 commented 1 year ago

It’s a 1.0 TB system and I’m pretty sure it’s a Polypolish issue. I was monitoring memory usage too and it wasn’t anywhere close. And I ran it with /usr/bin/time.

Can you tell me where to look?

On Jul 31, 2023, at 18:34, Ryan Wick @.***> wrote:

I've never tried Polypolish with such a large reference and read set, but based on that error, I think the most likely explanation is that you've run out of memory. How much RAM does your computer have?

You can check maximum memory usage with GNU Time https://www.gnu.org/software/time:

/usr/bin/time -v polypolish ... Then check Maximum resident set size (kbytes) after it finishes/crashes. Note that the exact location of time might depend on your OS.

— Reply to this email directly, view it on GitHub https://github.com/rrwick/Polypolish/issues/16#issuecomment-1659432600, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMXPRXC6C5NXIJOOQPAWUTXTBMJZANCNFSM6AAAAAA23BPIQU. You are receiving this because you authored the thread.

rrwick commented 1 year ago

Darn - if it's not running out of memory, then I'm not sure what the problem is. It seems to be crashing when running the load_assembly function, probably when building the per-base pileups.

The problem seems to occur with whatever contig is after q12_l5_m5_50397 - presumably q12_l5_m5_50398. Is that a particularly large contig or unusual in any way? If you delete that contig, does Polypolish work, or does it just crash on the next contig?

Sorry I can't be of more help. This case is outside of Polypolish's usual domain of bacterial isolates.

Ryan

tnn111 commented 1 year ago

You did mention it would be interesting to see what would happen with a eukaryotic genome. Now you know :-)

The largest contig is below 5 Mbases. I realize this is out of scope. I was hoping to use it to correct a large collection of metagenomes. We’re planning to burn through ~100 PromethION flow cells before the year is out. I’ll go stare at it and see if I can find something.

Thanks, Torben

On Jul 31, 2023, at 20:59, Ryan Wick @.***> wrote:

Darn - if it's not running out of memory, then I'm not sure what the problem is. It seems to be crashing when running the load_assembly function https://github.com/rrwick/Polypolish/blob/c04dae7bb5d1581f90948de33511d7fe32ed09fd/src/main.rs#L128-L140, probably when building the per-base pileups https://github.com/rrwick/Polypolish/blob/c04dae7bb5d1581f90948de33511d7fe32ed09fd/src/pileup.rs#L177-L187.

The problem seems to occur with whatever contig is after q12_l5_m5_50397 - presumably q12_l5_m5_50398. Is that a particularly large contig or unusual in any way? If you delete that contig, does Polypolish work, or does it just crash on the next contig?

Sorry I can't be of more help. This case is outside of Polypolish's usual domain of bacterial isolates.

Ryan

— Reply to this email directly, view it on GitHub https://github.com/rrwick/Polypolish/issues/16#issuecomment-1659528099, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMXPRS7IL2JFMGCGASKQV3XTB5KVANCNFSM6AAAAAA23BPIQU. You are receiving this because you authored the thread.

rrwick commented 1 year ago

Good luck - let me know if you find anything interesting!

rrwick commented 10 months ago

I'll close this issue because it's now a few months old. But please let me know if Polypolish is working for you or not on metagenomes.

By the way, I've just released Polypolish v0.6.0 which adds a --careful option which I recommend using for metagenomes.