Simpleneighbors for large indexes takes time to index using feed

aparrish / simpleneighbors

A clean and easy interface for performing nearest-neighbor lookups

BSD 3-Clause "New" or "Revised" License

50 stars 1 forks source link

Simpleneighbors for large indexes takes time to index using feed #9

Open thakur-nandan opened 4 years ago

thakur-nandan commented 4 years ago

Hi,

Thanks for the awesome package. I wanted to know if there is a way I could speed up the indexing of large multiple items (~500k - 1Mil) faster? Currently, the implementation works well, but it's really slow to index all the items. Any tips/tricks to faster the process?

Kind Regards, Nandan Thakur

aparrish commented 4 years ago

Unfortunately, the performance of both adding items to the index and building the index are limited by the upstream implementation of whatever backend you're using. If you're using the Annoy backend, you might pick through the Annoy GitHub issues page for hints and tips on speeding things up. Is it possible that your data set is too big to fit in RAM?