[FEA] API for index merges

rapidsai / cuvs

cuVS - a library for vector search and clustering on the GPU

https://rapids.ai

Apache License 2.0

206 stars 67 forks source link

[FEA] API for index merges #109

Open cjnolet opened 5 months ago

cjnolet commented 5 months ago

A common operation that's needed from some vector database architectures is the ability to merge two or more indexes into a single larger index. This is something that we should abstract behind a cuVS API so that we can continue to improve its performance over time.

(cc @benfred @tfeher @akiranaruse as I know we've discussed this in the past)

tarang-jain commented 4 months ago

FAISS has a CPU API for merging IVF indexes, wherein they simply add the IVF lists from one index to the other and reset the indices of the merged lists so that there are no repetitions. This is somewhat easy to implement for our IVF indices too. Will be more complicated to merge optimized CAGRA graphs though.