search

shuzhao-li-lab / asari

asari, metabolomics data preprocessing
Other
38 stars 9 forks source link

Asari Broken Pipe During Alignment #100

Open jmmitc06 opened 1 week ago

jmmitc06 commented 1 week ago

On most recent derivatized data sets an issue arises where the Asari process dies during alignment / mass grid construction. The error reported is a broken pipe from the multiprocessing pool; however the root cause is not immediately clear.

Some quick google searching suggests that the issue is related to memory consumption and use. in some cases, moving a large object through a pipe (evidently) causes it to crash / close. I believe that there is one or more samples whose peaks are too large to be serialized. The crash is reproducible, suggesting it is a particular sample that is a problem.

An initial workaround may be to have a try catch if the pipe breaks; however, the peak objects may need to be smaller in a future improvement or passed more compactly (perhaps the compressed pickle representation).

jmmitc06 commented 1 week ago

This sample broke this time: 

Cellpellets_Sample58_12C_DnHz_08132024_SZ_79