Semantic Search for the large corpora

[JoyeBright]

Greetings,

Just wanted to know what technique/ implementation you suggest for finding similar sentences over a very large corpus? 31M sentences, for instance!

Worth mentioning that my question is more concerned with memory management rather than speed.

Cheers, Javad

[aquibjaved]

Hey @JoyeBright

Now for similarity search you try the following options: If you want to implement from scratch you can use the following:

• Annoy https://github.com/spotify/annoy
• FAISS https://github.com/facebookresearch/faiss

Production ready pipelines you can use:

• Haystack https://github.com/deepset-ai/haystack

[nreimers]

Have a look at ANN here: https://www.sbert.net/examples/applications/semantic-search/README.html#approximate-nearest-neighbor