cholesky() function parallelization

[nfancy]

Describe your question-help A clear and concise description of what your question is or where do you want help.

Hi, Thank you very much for this great package. I want to use the cholesky() function in parallel mode. However, I don't see any parameter to control the cores used. Since the matrix that I am going to use is a really large one (30K*30K), I am worried, my sessions might crash if I don't control the cpu numbers and all cores are used. Please advise what's the best approach. Thanks in advance.

Best wishes Nurun

[ManosPapadakis95]

I don't believe that your sessions are going to crash. Maybe your computer freeze but depends on your device. Can you tell me operating system, ram, CPU characteristics?

The reason I haven't add a variable for controlling the number of threads is because i disagree with that option. Anyway I might add it in a future version because a lot of people ask for it.

[nfancy]

Hi, Thank you very much for your quick response. I tried with a matrix of dimension 60K x 60K and the Rsession crashed, then I tried with 30K x 30K (downsampled to half) and it ran fine. My machine has 64 cores and 264GB RAM. My system conf is as follows:

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              64
On-line CPU(s) list: 0-63
Thread(s) per core:  2
Core(s) per socket:  32
Socket(s):           1
NUMA node(s):        1
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               49
Model name:          AMD Ryzen Threadripper 3970X 32-Core Processor
Stepping:            0
CPU MHz:             2191.182
CPU max MHz:         3700.0000
CPU min MHz:         2200.0000
BogoMIPS:            7386.45
Virtualisation:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-63

Any suggestion is hugely appreciated.

[ManosPapadakis95]

Please try again with 60k to see if the crashes continue. Sometimes R is crashing for no reason...

[wmacnair]

Hi, I have a similar question.

I would like to run Rfast::colsums in parallel on a large cluster, with control over how many cores I use. What would you recommend?

You said "The reason I haven't add a variable for controlling the number of threads is because i disagree with that option.". I disagree with your disagreement ;) Sometimes it's important for users to be able to be able to guarantee how many cores are going to be used by a process, so that the administrators of their cluster don't get annoyed with them.

Thanks Will

[ManosPapadakis95]

Hi will,

I fully understand what you are saying but my problem about the number of cores as option it has to do with one specific reason, nobody knows what that means and they annoying me instead. I should have add an option for choosing the number of cores with default value to 1. Some functions cannot be fast unless they use all the cores! I had to explain that to each person separately but they keep telling me that the parallel is not fast enough. I know that and you can fix it by using all the cores. But know every ones asking me why we have to use all the cores. So, I decide to add an option to every parallel algorithm. I cannot deal with that now because I have just updated Rfast and I have to update Rfast2 too. It will take some time although because I have to change 100+ functions. I haven't any recommendations.

Manos