search

egeulgen / pathfindR

pathfindR: Enrichment Analysis Utilizing Active Subnetworks
https://egeulgen.github.io/pathfindR/
Other
178 stars 25 forks source link

n_processes and iterations (How to block memory/CPU in the cluster) #146

Closed Nitin123-4 closed 1 year ago

Nitin123-4 commented 1 year ago

Hi I am running pathfindR with default parameters with 10 iterations and it is taking 10 n_processes as n_processes was set to NULL.

Can you please help how many CPUS and memory I should block for this process in the cluster?

If I will block 20 CPUs and 35 GB space looks like it is failing in-between because the java code is creating 10 processes and it is stopping by the scheduler.

Please help with it.

egeulgen commented 1 year ago

Hello,

The java code runs on a single thread for each iteration. For executing each iteration in parallel, a foreach loop is used. One option to overcome your issue would be running the workflow manually, which is fairly simple and described here.

I wouldn't say I'm very knowledgeable to advise on blocking resources on a cluster but I would block 10 CPUs but am not sure about memory. With the example input that we provide (which would be a typical input size), the memory requirement does not seem to exceed 1GB.

Hope this helps, Best, -E