opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data
https://opensearch.org/docs/latest/search-plugins/knn/index/
Apache License 2.0
156 stars 123 forks source link

Introduce native Readers read flatValues directly from faiss file. #2267

Open luyuncheng opened 1 week ago

luyuncheng commented 1 week ago

ISSUE: #2266

Faiss code: faiss/impl/index_write.cpp


|-typeIDMap   -|-id_header-|
|-typeHnsw    -|-hnsw_header-|-hnswGraph-|
|-typeStorage -|-storage_Header-|-storageVector-|
|-idmap_vector-|-FOOTER_MAGIC+CHECKSUM-|

AND IndexHeader like:

|dim|ntotal|dummy|dummy|is_trained|metric_type|metric_arg|

Example for HNSW32,Flat, the whole file like:

|idMapType|idMapHeader|hnswType|hnswHeader|hnswGraph|flatType|flatHeader|Vectors|IdVector|FOOTER_MAGIC+CHECKSUM|

so we can read the file from directly like this PR