search

open2c / cooler

A cool place to store your Hi-C
https://open2c.github.io/cooler
BSD 3-Clause "New" or "Revised" License
208 stars 50 forks source link

Cooler zoomify issues summary thread #232

Open mimakaev opened 3 years ago

mimakaev commented 3 years ago

On Open2C there have been several reports about cooler zoomify misbehaving in a non-reproducible fashion. Using this as a meta-thread summarizing our observations.

@sergpolly and @Phlya reported that cooler zoomify may take too much RAM, but I'm not sure if this could be related to this issue. I did some testing and did not detect memory leaks, although noted very large RAM usage even when reducing default chunk size from 10M to 2.5M (memory went down from 2G to 1.5G for main process).

I see that "cooler zoomify" sometimes stops after the first round of results. I have seen it happen several times at 1000 -> 5000 zoomify. Below is the command output from distiller run. It is not a RAM issue in my case as the machine does not have any memory limits. Restarting the jobs nearly always makes it complete for this type of issues for me.

image

moxqiang commented 1 year ago

I'm having the same problem as you. my code（cooler 0.8.6）： cooler zoomify -p 20 -r 5000,10000,25000,50000,100000,250000,500000,1000000,2500000,5000000,10000000 -o /public/home/qqfu/HPV-PoreC/HiPore-C/Use.hg38_hpv16_18.Res-02/Result/Cooler/SiHa.1000.mcool --balance SiHa.1000.cool

I found that I was fine without the -p parameter