facebookresearch / faiss

A library for efficient similarity search and clustering of dense vectors.
https://faiss.ai
MIT License
30.93k stars 3.61k forks source link

FAISS Index Factory String equivalent code and vice versa #3843

Closed ashleyabraham closed 1 month ago

ashleyabraham commented 1 month ago

Is the FAISS Index Factory string IVF132_HNSW32,PQ5x8

D = 128
index = faiss.index_factory(D, `IVF132_HNSW32,PQ5x8`)

equivalent to the following code

hnsw_m = 32
nlist = 132
M=5
nbits = 8

quantizer = faiss.IndexHNSWFlat(D, hnsw_m)
index = faiss.IndexIVFPQ(quantizer, D, nlist, M, nbits)

If not what is the correct code for the above index factory string and also what is the index factory string for the above code

mengdilin commented 1 month ago

Yes they are equivalent. The issue here is faiss expects dimension has to be a multiple of M (# of subquantizers). Right now, you set the M to 5 and 128 is not divisible by 5. An example fix would be setting M to 4

D = 128
index = faiss.index_factory(D, "IVF132_HNSW32,PQ4x8")
ashleyabraham commented 1 month ago

Thank you!