Figure out how to deal with large-scale dataset

[ebursztein]

We can do our own using a gpu accelerate lib

• https://medium.com/rapids-ai/scaling-knn-to-new-heights-using-rapids-cuml-and-dask-63410983acfe
• https://github.com/facebookresearch/faiss
• https://github.com/yahoojapan/NGT (vlad algorithm)

serving

• https://oss.redislabs.com/redisai/intro/

or use a fully fledge server

• https://github.com/vdaas/vald (NGT)
• https://github.com/milvus-io/milvus (FAISS)

[xiaofan-luan]

Hey @ebursztein , any progress on this issue? we can help on integrate milvus together with tf similarity if we decide to take fully fledge server solution