Computation on CondMat_network example takes too much time

[sonjageorgievska]

With 4 threads and alpha = 4 it takes 80 minutes. My machine has 2 cores times 2 GHz and 16 GB RAM. The process used 428 MB and CPU utilization was between 90% and 100% all the time.

Has anyone else had this problem? Or any insights?

[DataWaveAnalytics]

Are you sure you can run 4 threads on your machine? However, the memory consumption shouldn't be that high. I will do some tests.

If you are in linux, run lscpu. Then to obtain the total number of threads consider the nex info: Threads(s) per core: 2 Core(s) per socket: 4 Socket(s): 1

In this example, you can run max. 2x4x1 = 8 threads

[DataWaveAnalytics]

Are you using the python wrapper for your test?

[DataWaveAnalytics]

In my machine, 8 x Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60 GHz CondMat_network takes (Note: I used the C++ implementation directly)

2 threads: ~16m 27s 4 threads: ~8m 12s 8 threads: ~5m 33s

All of them using 391mb RAM. The algorithm is CPU intensive, so it is OK a 90-100% CPU usage.

[sonjageorgievska]

Thanks, the numbers that you gave in the last comment are comparable to mine, when taking into account number of cores/processor_speed/ number_of_threads.

For the other questions: I used the c++ implementation directly (no python wrapper). The first comment was not clear from beginning to end, but in principle you can run a large number of threads on any machine (unless limited by application). However, after a certain number of threads the performances will degrade due to ''context switch'' time. In my case with 2 threads the CPU was under-utilized so I tried 4 and it was fully utilized (and the program ran faster).

[sonjageorgievska]

I will close this for now, as it is not really an issue. It is more important that it is efficient on large datasets (compared to bh-tSNE).