search

wangjr03 / FLAMINGO

MIT License
14 stars 10 forks source link

Memory usage is much higher than described in the paper #11

Open sXperfect opened 1 year ago

sXperfect commented 1 year ago

Hi,

I tried to reproduce the FLAMINGO result using .hic data described in the readme and reconstructed the structure of Chromosome 1. But it turns out FLAMINGO used up to 80GiB of memory. I also tried on a PC with 64 GiB of RAM and it always crash.

The screenshot of the memory usage can be seen here: Screenshot 2022-12-22 190158

Which setting should i change to reduce the memory peak?

-sXperfect

SarahBeecroft commented 1 year ago

I found I had more reliable performance when reducing n_rows wayyy down to like 300. I did also find the mem usage was much higher than described and it kept crashing my system until I reduced n_rows. I hope that helps :)

JiaxinYangJX commented 12 months ago

Thanks for using our software! The space complexity is O(kN^2), where N is the number of points of the low-resolution backbone or the number of points in each high-resolution domain. For example, we reconstructed 1Mb backbone and 5kb domain structures in chr1, thus the number of points of the backbone is around 280 and the number of points in each high-resolution domain is 200.

Could you check how many points are in the reconstruction task? Balancing the low-resolution and high-resolution parameters is a good idea to avoid extremely large numbers of points in a single task.

In addition, we are currently collecting all the feedback and reorganizing our tools. We will release a more user-friendly version by next month.

sXperfect commented 7 months ago

Thanks for using our software! The space complexity is O(kN^2), where N is the number of points of the low-resolution backbone or the number of points in each high-resolution domain. For example, we reconstructed 1Mb backbone and 5kb domain structures in chr1, thus the number of points of the backbone is around 280 and the number of points in each high-resolution domain is 200.

Could you check how many points are in the reconstruction task? Balancing the low-resolution and high-resolution parameters is a good idea to avoid extremely large numbers of points in a single task.

In addition, we are currently collecting all the feedback and reorganizing our tools. We will release a more user-friendly version by next month.

Is there any update regarding this one?

So the problem i mentioned at the very beginning is using the default parameters described in the README. Could you perhaps list the parameter values to reproduce the results in the paper?

JiaxinYangJX commented 7 months ago

Hi @sXperfect @SarahBeecroft Thanks for using our tools! We just released a lite version of FLAMINGO, which is faster, more memory-efficient, and more user-friendly. We fixed the majority of bugs and hope the new version could help. Below is the link. https://github.com/JiaxinYangJX/FLAMINGOrLite