search

deeptools / HiCExplorer

HiCExplorer is a powerful and easy to use set of tools to process, normalize and visualize Hi-C data.
https://hicexplorer.readthedocs.org
GNU General Public License v3.0
233 stars 70 forks source link

hicNormalize is really slow #671

Closed cgirardot closed 3 years ago

cgirardot commented 3 years ago

Dear, we are trying to run hicNormalize 3.6 on 6 h5 matrice (1K bins, Dm6 genome, 170-340 M in size) but it is really slow and is running since 4 days and uses like 3% of a CPU.

So we moved to a recent server with a lot of memory, copied the smallest matrix (170M) to the local SSD disk, and tried

hicNormalize -m matrix.h5 -n multiplicative -o matrix.NormSmallest.h5 -mv 0.9476211335

the process is running like 2% of a CPU, is this normal?

joachimwolff commented 3 years ago

Hi,

2% CPU usage is indeed not much, however, this number can be correct. How many cores / threads does your system have and how does your system measures the percentage value? In my experience two options are common: Either all threads together count as 100%, or each thread count itself as 100% and the accumulation for the system is then e.g. for 16 threads 1600%.

Anyway, for me, this sounds more like something crashed. Can you tell me how much memory is available and can you observe how CPU and memory usage changes over time? If there is a crash, the CPU usage should be high for one thread, and at some point, it drops and stays at the level you described.

Best,

Joachim

cgirardot commented 3 years ago

We have 64 cores and the percentage I am giving you is the CPU usage of the hicNormalize process only, on this screenshot, we see that threads using more than one CPU do appear as 363%.

Screenshot 2021-02-10 at 12 12 12

The hicNormalize is one of these "2%" down the list, running for 20h now:

Screenshot 2021-02-10 at 12 17 29

The server has 2Tb of memory, plenty of which is available. We tried on 2 different servers. I am not sure to follow your crash explanation. Are you suggesting to restart and monitor the CPU/memory usage?

joachimwolff commented 3 years ago

Yes, something is really wrong. Please cancel the process, restart it and monitor the usage as I suggested. If there is a high usage in the beginning but drops down to these 2% there is a high possibility to have an undetected bug.

cgirardot commented 3 years ago

OK will do, I can also try with other versions. Any suggestion on an earlier version that you know works for sure ?

joachimwolff commented 3 years ago

This version worked so far fine and is a single-core implementation. All we do is a multiplication of the csr_matrix.data array with your multiplicative input value. I am scratching my head atm what could be the issue, earlier version had either not this feature or if they had it, it was not changed in the recent versions.

cgirardot commented 3 years ago

oh wait, I think there is something weird with the bash script that executes the command. I just tried another test, directly from th command line and it worked in 10s...

cgirardot commented 3 years ago

OK my deepest apologies for wasting your time. I was not running the command line myself and I am not sure what the user did exactly. All my tests look OK now. I should have tried myself before posting! Sorry again