search

krishnanlab / PecanPy

A fast, parallelized, memory efficient, and cache-optimized Python implementation of node2vec
BSD 3-Clause "New" or "Revised" License
155 stars 22 forks source link

Gensim word2vec can not scale to networks with million nodes efficiently, find alternative #19

Open RemyLau opened 3 years ago

ChristopherMancuso commented 2 years ago

@RemyLau I was building some of the cross species networks and I noticed that as the threads were increased (I have got up to 120 cpus going), generating the walks became very fast, however, the training time of the gensim model slows down a considerable amount. Using too many threads in gensim seems to be a known problem as shared in this post.. I think there could be two ways to help fix this

  1. The easy fix would be too add separate arguments for the number of workers in random walk generation and the number of workers used by gensim and let the user find what is best
  2. The other could be to use the corpus_file argument as described in the post above. Maybe it would be fast with large parallel processing to generate the corpus_file, and if users have some scratch system like MSU could easily be saved there temporarily
RemyLau commented 2 years ago

@ChristopherMancuso thanks for the suggestion! I'll try to do some testing and see how it goes. From my experience, gensim don't work well on AMD machines. But on Intel's, it's pretty fast and scales pretty well; not sure if there's architectural optimization done specifically for intel's chip.

I will transfer your reply to a new issue to keep things separate.