Closed Ishitori closed 1 year ago
Hi!
What is the output of infomap --version
?
If it doesn't say that it is compiled with OpenMP support, you might be missing the libomp-dev
package.
How big is the network in terms of file size? Is it an ordinary or multilayer network? Does it include node names?
If you are adding the network in Python by looping over the links, this is significantly slower than reading a file directly with im.read_file("network.net")
.
How much memory does your machine have? If you're running out of memory, you might be using swap instead of ram, which will be much slower.
The output of infomap --version
is:
Infomap version 2.6.1 compiled with OpenMP
So, it looks like it has OpenMP support
My network is about 200M nodes and 1B edges. It is ordinary, directed, weighted network. It doesn't include node names, all nodes are encoded with a consecutive integers from 0 to N. It takes about 13 minutes to add all the links from parquet files to infomap - slow but not terrible.
I am running it on a huge machine with 512 gigs of RAM. According to top
the process uses 63.3% of RAM only.
It is running for the 4th day in a row as of now. I was thinking that maybe it is doing some single-thread preprocessing before starting working at full speed, but no, I don't see a switch to multi-core usage.
Any recommendations?
Alright!
Yes, Infomap runs the PageRank algorithm single-threaded, as this step is usually significantly faster than the optimization step, where we use OpenMP.
Sounds like you have done everything right, so my suggestions is to check out these alternative Infomap implementations: Scalable and Efficient Flow-Based Community Detection for Large-Scale Graph Analysis Multi-level Graph Drawing using Infomap Clustering
I'm also converting this to a discussion for now!
Hi team,
I have a 200M node graph and Ubuntu with multiple cores. It takes about 2.5 hours to load the graph, but to my surprise things are still slow even when running
im.run()
- according tohtop
only one core of my machine is used... I thought infomap is written with OpenMP support... I installed using regularpip install
.What should I do to run infomap in parallel?