search

songjiajia2018 / LINEAGE

LINEAGE
GNU General Public License v3.0
8 stars 0 forks source link

Help on performance #1

Open lconan opened 1 year ago

lconan commented 1 year ago

Hi, really appretiate the tool to do lineage determination with mitochondrial variants. I've been running lineage with 8 cores, and I noticed that it nearly consumes 100G ram per thread, and it's been running for 97 hours. Wondering if anything could go wrong that I can check or just terminate it? I called the variants with mgatk, and got the 3.8MB matrix (gzipped) with ppl. Thanks a lot.

linl-linl commented 1 year ago

Did you run LINEAGE on high throughput scRNA-seq data from 3’/5’-end-directed scRNA-seq technologies? LINEAGE is designed for full-length scRNA-seq, and the computation time and memory requirement will increase with the number of cells. So, we suggest performing LINEAGE on scRNA-seq data with less than 1000 cells.

andrecossa5 commented 1 year ago

Hi, and thank you again for the great tool. I am facing parallelization and memory issues when using the lineage function. My data consist of an allele frequency matrix of ~500 cells. Calling the lineage function (with both the multi- and single-core behaviour) for even a small number of iterations (i.e., 2) results in all CPUs cores (32 cores, 129GB memory) completelly filled up. I could not see any output result, because eventually all processes crash. May I ask you for any usage suggestion? I am setting up a benchmark study, so having your method will be a very nice thing. Thank you in advance!

linl-linl commented 1 year ago

Hi, this is likely due to insufficient memory. Considering the memory you have, you can run LINEAGE on smaller data to get the output.


---- Replied Message ----
From ***@***.***>
Date 06/07/2023 01:10
To ***@***.***>
Cc ***@***.***>、***@***.***>
Subject Re: [songjiajia2018/LINEAGE] Help on performance (Issue #1)

Hi, and thank you again for the great tool. I am facing parallelization and memory issues when using the lineage function. My data consist of an allele frequency matrix of ~500 cells. Calling the lineage function (with both the multi- and single-core behaviour) for even a small number of iterations (i.e., 2) results in all CPUs cores (32 cores, 129GB memory) completelly filled up. I could not see any output result, because eventually all processes crash. May I ask you for any usage suggestion? I am setting up a benchmark study, so having your method will be a very nice thing. Thank you in advance!

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: <songjiajia2018/LINEAGE/issues/1/1579152194@github.com>

andrecossa5 commented 1 year ago

Hi there,

Thanks for your reply. Since we have 500 cells, which is below the suggested limit (1000 cells), can this be because I am occupying all the available CPUs? If yes, can you clarify whether there was a way to control the number of CPUs used? Even specifying repeats=2 and threads=1 to the lineage function resulted in the whole occupancy of the CPUs.

Thanks in advance for your help Best Andrea