Atropos can hang in multi-threaded mode due to insufficient memory

jdidion / atropos

An NGS read trimming tool that is specific, sensitive, and speedy. (production)

Other

120 stars 15 forks source link

Atropos can hang in multi-threaded mode due to insufficient memory #4

Closed jdidion closed 6 years ago

jdidion commented 7 years ago

This is a subtle bug in which the main thread sometimes hangs waiting to read the next record from the input file. This appears to occur only under a strictly-regulated memory cap such as on cluster environment.

To address this, I have so far added the following:

Set the default batch size based on the queue sizes
Warn the user if their combination of batch and queue sizes might lead to excessive memory usage

jdidion commented 6 years ago

Closing this. Memory usage will be substantially improved on py37+, with the use of the new gc.freeze() method.