search

nanoporetech / pod5-file-format

Pod5: a high performance file format for nanopore reads.
https://pod5.nanoporetech.com
Other
121 stars 15 forks source link

Guppy v6.5.7 with pod5 uses tons of memory vs fast5 #45

Closed billytcl closed 1 year ago

billytcl commented 1 year ago

I converted fast5s to a ~900 GB single pod5 file (v0.2). When basecalling, it is taking up a ton of memory:

image

This doesn't happen when I call from fast5s. Is this normal?

0x55555555 commented 1 year ago

Hi @billytcl,

This is due to how pod5 uses virtual memory, it's to do with how data is cached in case its needed again.

Does the basecall run ignoring the high memory use?

We will look at changing config to avoid this memory usage.

Thanks,

billytcl commented 1 year ago

As long as it won't crash the system then it's all good!

On Mon, Jun 12, 2023 at 1:01 AM jorj1988 @.***> wrote:

Hi @billytcl https://github.com/billytcl,

This is due to how pod5 uses virtual memory, it's to do with how data is cached in case its needed again.

Does the basecall run ignoring the high memory use?

We will look at changing config to avoid this memory usage.

Thanks,

  • George

— Reply to this email directly, view it on GitHub https://github.com/nanoporetech/pod5-file-format/issues/45#issuecomment-1586790748, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPHYT24NG3JOVQBP2KGGZDXK3EFBANCNFSM6AAAAAAZC4OMK4 . You are receiving this because you were mentioned.Message ID: @.***>

0x55555555 commented 1 year ago

Hi @billytcl,

It's OS managed, so will be released once another process needs the memory. It should be harmless, but I agree it can look worrying.