Large Graph fills almost complete memory

[Engineerumair]

The following code almost takes all of the memory. node2vec = Node2Vec(G_fb, dimensions=emb_size, walk_length=80, num_walks=10, workers=4)

I have 12gb Memory and the nodes are 1.6 million which takes all of the memory and I have processed only 0.1 million node and then got memory error.

Here is the error:

---

MemoryError Traceback (most recent call last)

in () 4 5 for emb_size in [32, 128]: ----> 6 node2vec = Node2Vec(G_fb, dimensions=emb_size, walk_length=80, num_walks=10, workers=3) # Use temp_folder for big graphs 7 model = node2vec.fit(window=10, min_count=1,batch_words=4) 8 model.wv.save_word2vec_format("mypath/%s_%d.deepwalk"%(graph_name,emb_size))

...site-packages\node2vec\node2vec.py in init(self, graph, dimensions, walk_length, num_walks, p, q, weight_key, workers, sampling_strategy, quiet, temp_folder) 68 self.require = "sharedmem" 69 ---> 70 self._precompute_probabilities() 71 self.walks = self._generate_walks() 72

...site-packages\node2vec\node2vec.py in _precompute_probabilities(self) 117 if current_node not in first_travel_done: 118 first_travel_weights.append(self.graph[current_node][destination].get(self.weight_key, 1)) --> 119 d_neighbors.append(destination) 120 121 # Normalize

MemoryError:

Is there a way to process some of the nodes and then save the embedding file and then again process some of the nodes and append those to the saved file and so on?

[SihamAmarouche]

I have the same problem.Had it solved?

[Engineerumair]

I have the same problem.Had it solved?

No, it is not solved yet

[worthless443]

could this be that it is not a bug and that it needs just more memory?

[SR023]

use randomwalk and save file to local, then use fasttext get vec.