facebookresearch / faiss

A library for efficient similarity search and clustering of dense vectors.
https://faiss.ai
MIT License
30.35k stars 3.54k forks source link

Writing and reading IndexIDMap (C++) #3647

Closed frootonator closed 1 month ago

frootonator commented 1 month ago

Is there a correct way to write and read IndexIDMap? Both methods require data type faiss::Index*.

I've tried to do it this way:

faiss::write_index(dynamic_cast<faiss::Index *>(indexIDMap), "path.index");
faiss::Index * index = faiss::read_index("path.index");
indexIDMap= reinterpret_cast<faiss::IndexIDMap *>(index);

But I'm 99.9% sure it's incorrect.

mnorris11 commented 1 month ago

IndexIDMap is used to enable add_with_ids on indexes that do not support it, like the Flat indexes. It wraps some other index. Then the vectors are stored on that other underlying index.

https://github.com/facebookresearch/faiss/wiki/Pre--and-post-processing#the-indexidmap

Is this along the lines of what you are looking for?

frootonator commented 1 month ago

IndexIDMap is used to enable add_with_ids on indexes that do not support it, like the Flat indexes. It wraps some other index. Then the vectors are stored on that other underlying index.

https://github.com/facebookresearch/faiss/wiki/Pre--and-post-processing#the-indexidmap

Is this along the lines of what you are looking for?

In this case I can write, for example, IndexHNSWFlat that was wrapped by IndexIDMap.

faiss::IndexIDMap index = faiss::IndexIDMap(indexHNSW);
faiss::write_index(dynamic_cast<faiss::Index *>(indexHNSW), "path.index");

But then I still have to read it as usual index...

faiss::Index * index = faiss::read_index("path.index");
faiss::IndexHNSWFlat * indexHNSW = reinterpret_cast<faiss::IndexHNSWFlat *>(index);

And in this case it's impossible to encapsulate it by IndexIDMap because wrapped index must be empty. So I still have to cast it somehow to IndexIDMap indirectly. Feels incorrect.