Parallel Computation

[pasinit]

Hi, i can't figure out if i can compute personalized pagerank for different nodes in parallel. So my question is if personalized_pagerank is thread safe method or i have to replicate the graph for each thread i want to use.

Thanks.

[ntamas]

igraph is not thread-safe - actually, the underlying C library is not thread-safe, even if you replicate the graph for each thread. The reason is that the underlying C library has a global, shared stack where it pushes the pointers to each memory area that is allocated by some function in order to be able to deallocate these memory blocks when an error occurs anywhere during the computation. Since this stack is shared between threads, a computation failure in one thread would cause the entire stack to roll back.

The C core of igraph could in theory be compiled in thread-safe mode - there is a flag that could be passed to the ./configure script when the C core is compiled. In this case, the global variables (the error stack and similar constructs) are replaced by thread-local ones. However, this has not been tested thoroughly, especially not in the context of the Python interface, so I would strongly advise against it.

Since the personalized PageRank computation is a CPU-intensive task, you could probably achieve almost the same performance by splitting the computation between multiple processes. The multiprocessing module in Python is the standard tool for this, but I've never tried multiprocessing with igraph so I don't know whether an igraph Graph can be handled by multiprocessing. However, in the worst case, you can launch multiple workers using multiprocessing, then make each worker load its own copy of the graph after it has started.

[pasinit]

Thanks for the fast and exhaustive answers, i think i'll proceed using multiprocessing as you suggested! Thanks a lot again!!

[ntamas]

Let me know how it went or feel free to ask if you are running into issues.