search

petecheng / Time2Graph

Source codes for Time2Graph model.
133 stars 33 forks source link

MemoryError #1

Closed reborm closed 3 years ago

reborm commented 3 years ago

Thank you for sharing you code. I found that when I used larger graph data, the MemoryError occured. Could you please help me?

Number of nodes: 183209 Number of walks: 1832090 Data size (walks*length): 73283600 Walking... Traceback (most recent call last): File "/search/odin/xxx/anaconda3/envs/push_py27/bin/deepwalk", line 11, in load_entry_point('deepwalk==1.0.3', 'console_scripts', 'deepwalk')() File "/search/odin/xxx/anaconda3/envs/push_py27/lib/python2.7/site-packages/deepwalk-1.0.3-py2.7.egg/deepwalk/main.py", line 151, in main process(args) File "/search/odin/xxx/anaconda3/envs/push_py27/lib/python2.7/site-packages/deepwalk-1.0.3-py2.7.egg/deepwalk/main.py", line 80, in process walks = weighted_random_walk.random_walk(G, num_paths=args.number_walks,path_length=args.walk_length, alpha=0) File "/search/odin/xxx/anaconda3/envs/push_py27/lib/python2.7/site-packages/deepwalk-1.0.3-py2.7.egg/deepwalk/weighted_random_walk.py", line 42, in random_walk transitionMatrix = getTransitionMatrix(G, nodes) File "/search/odin/xxx/anaconda3/envs/push_py27/lib/python2.7/site-packages/deepwalk-1.0.3-py2.7.egg/deepwalk/weighted_random_walk.py", line 6, in getTransitionMatrix matrix = np.zeros([len(nodes), len(nodes)]) MemoryError

petecheng commented 3 years ago

This error is raised probably because a transition matrix with size of 180K * 180K fails to be allocated, since it is too large for your current memory. And in the framework of Time2Graph, the hyperorarameter K (number of extracted shapelets) is usually small, in other words, the size of the shapelet graph is small enough such that the abovementioned error would not occur in most cases. If you just want to apply deepwalk on a large graph, you may need to adopt some other other scalable and efficient implementations of this graph embedding algorithm.

reborm commented 3 years ago

This error is raised probably because a transition matrix with size of 180K * 180K fails to be allocated, since it is too large for your current memory. And in the framework of Time2Graph, the hyperorarameter K (number of extracted shapelets) is usually small, in other words, the size of the shapelet graph is small enough such that the abovementioned error would not occur in most cases. If you just want to apply deepwalk on a large graph, you may need to adopt some other other scalable and efficient implementations of this graph embedding algorithm.

Ok, thank you for your reply!!!