facebookresearch / faiss

A library for efficient similarity search and clustering of dense vectors.
https://faiss.ai
MIT License
31.63k stars 3.65k forks source link

Refine Layer not support `add_with_ids` #3820

Closed Tickdack closed 2 months ago

Tickdack commented 2 months ago

Summary

I want to use refine layer to re-order the coarse-results for more accurate output, but I find refine-layer is seemingly not support add_with_ids, I'm a little confused because the raw id should be stored in coarse-grained sorting model, the refine layer just rerank the coarse-result. How can I modify my parameters to use a refine-layer that support add_with_ids for improve the accuracy? Do I have to use IDMap in refine-layer? I think it's cracy to memory.

Platform

OS: CentOS

Faiss version: faiss 1.5

Running on:

Interface:

Reproduction instructions

Here is my index_type parameter: IVF20480,PQ192x4fs,Refine(PQ96x8)

Tickdack commented 2 months ago

I read the source Cpp file, and I find the class IndexRefine, I find IndexRefine not support add_with_ids, but I want to use fastscan index(Refine is necessary), what the general scheme to use it if i want to get our own assigned id when search?thank you.

mdouze commented 2 months ago

The IndexRefine does not support add_with_ids because the ids need to be sequential indices for the refinement index, which is most often an IndexFlatCodes. If you need add_with_ids, please wrap the index in an IndexIDMap.

Tickdack commented 2 months ago

The IndexRefine does not support add_with_ids because the ids need to be sequential indices for the refinement index, which is most often an IndexFlatCodes. If you need add_with_ids, please wrap the index in an IndexIDMap.

Thank you sir, it means I should use "IDMap,IVF20480,PQ192X4fsr,Refine(PQ96x8)" as the index factory key, is that right?

Tickdack commented 2 months ago

The IndexRefine does not support add_with_ids because the ids need to be sequential indices for the refinement index, which is most often an IndexFlatCodes. If you need add_with_ids, please wrap the index in an IndexIDMap.

read the function index_factory and cpp files, i find refine also not support remove_ids, remove_ids on the index generated by parameter "IDMap,IVF20480,PQ192X4fsr,Refine(PQ96x8)" is not work, too. i think fastscann is a potential index to our project, how can i use it with high-performance accuracy and inner store and remove our assigned ids?

mdouze commented 2 months ago

Right, supporting revove_ids would be an enhancement. Making a new issue for this.