search

snap-stanford / snap

Stanford Network Analysis Platform (SNAP) is a general purpose network analysis and graph mining library.
Other
2.17k stars 797 forks source link

node2vec on a 1e9 edges network: edge reading stopped and return `Killed` #142

Closed marcevrard closed 6 years ago

marcevrard commented 6 years ago

I tried to run the SNAP version of node2vec installed on Ubuntu 14.04 desktop with 10 core double threaded (20) and 128 GB of ram. The edgelist file is about 20GB containing about 1e9 edges and 1e7 vertices (expressed in integers); the reading takes about 20h then stopped and return Killed (after reporting the correct count of edges). The RAM usage is still far from the desktop limit moments before the stopping.

Is this graph size above what node2vec can handle?

zhushun0008 commented 6 years ago

I have the same issue @marcevrard . LINE algorithm could handle 1e9 size, but I did not install it successfully.

roks commented 6 years ago

This is a large graph, so memory requirements likely exceed your RAM. Memory usage depends highly on edge distribution. To find the pretty accurate lower bound, use function PredictMemoryRequirements in snap-adv/biasedrandomwalk.cpp

clintonthomas commented 4 years ago

你的内存太小了