alipay / vsag

vsag is a vector indexing library used for similarity search.
Apache License 2.0
155 stars 14 forks source link

Add a new Layer "DataCell" to manage data access #40

Open LHT129 opened 1 month ago

LHT129 commented 1 month ago

I think high level index should ignore the specific implement of quantization and io type, So the quantization and io components can be managed by new layer DataCell which provide different data access interface to the index layer. just like the following show

Layer Description Implement
Index Layer ANN algorithm & architecture. Graph (HNSW,NNDescent) + Reorder, IVF(PQ,SQ,fastscan), HNSW_IVF, MultiIndex ......
DataCell Layer data access, use quantization, io and others FlattenDC, GraphNeighborhoodDC, MixtureDC .......
Operator Layer quantizations, IO and other basic components PQ,SQ,MemoryIO,SSDIO,......

this abstract layered approach may facilitate scalability and code reuse currently we only have index layer, so the issue will provide Datacell Layer and Operator Layer in this feature. the related pull requests:

LHT129 commented 1 week ago

Maybe like this

image