Closed Tickdack closed 2 months ago
I read the source Cpp file, and I find the class IndexRefine
, I find IndexRefine
not support add_with_ids
, but I want to use fastscan
index(Refine is necessary), what the general scheme to use it if i want to get our own assigned id when search
?thank you.
The IndexRefine
does not support add_with_ids
because the ids need to be sequential indices for the refinement index, which is most often an IndexFlatCodes
.
If you need add_with_ids
, please wrap the index in an IndexIDMap
.
The
IndexRefine
does not supportadd_with_ids
because the ids need to be sequential indices for the refinement index, which is most often anIndexFlatCodes
. If you needadd_with_ids
, please wrap the index in anIndexIDMap
.
Thank you sir, it means I should use "IDMap,IVF20480,PQ192X4fsr,Refine(PQ96x8)" as the index factory key, is that right?
The
IndexRefine
does not supportadd_with_ids
because the ids need to be sequential indices for the refinement index, which is most often anIndexFlatCodes
. If you needadd_with_ids
, please wrap the index in anIndexIDMap
.
read the function index_factory
and cpp files, i find refine
also not support remove_ids
, remove_ids
on the index generated by parameter "IDMap,IVF20480,PQ192X4fsr,Refine(PQ96x8)" is not work, too. i think fastscann
is a potential index to our project, how can i use it with high-performance accuracy and inner store and remove our assigned ids?
Right, supporting revove_ids
would be an enhancement. Making a new issue for this.
Summary
I want to use refine layer to re-order the coarse-results for more accurate output, but I find refine-layer is seemingly not support add_with_ids, I'm a little confused because the raw id should be stored in coarse-grained sorting model, the refine layer just rerank the coarse-result. How can I modify my parameters to use a refine-layer that support
add_with_ids
for improve the accuracy? Do I have to useIDMap
in refine-layer? I think it's cracy to memory.Platform
OS: CentOS
Faiss version: faiss 1.5
Running on:
Interface:
Reproduction instructions
Here is my index_type parameter:
IVF20480,PQ192x4fs,Refine(PQ96x8)