search

opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data
https://opensearch.org/docs/latest/search-plugins/knn/index/
Apache License 2.0
156 stars 123 forks source link

Add IVF changes to support Faiss byte vector #2002

Closed naveentatikonda closed 2 months ago

naveentatikonda commented 2 months ago

Description

Add IVF changes to support Faiss byte vector which behind the scenes uses Faiss SQ8_direct_signed scalar quantizer.

Related Issues

https://github.com/opensearch-project/k-NN/issues/1659

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

navneet1v commented 2 months ago

@naveentatikonda can we put a small benchmark run for this to showcase that ByteVector with IVF is working and we are seeing drop in memory usage.

naveentatikonda commented 2 months ago

@naveentatikonda can we put a small benchmark run for this to showcase that ByteVector with IVF is working and we are seeing drop in memory usage.

This is the RSS metrics for Cohere-wiki-768 dataset (475,858 vectors) and the memory usage is as expected. IVF memory requirement - 0.4 gb (appx.) Heap - 1GB Total memory provided - 1.7 GB

image