search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.66k stars 1.04k forks source link

When will index AMs such as `hnsw` and `ivfflat` be added to vector indexes for external use to fully enable end-to-end support for vector indexes?[yugabyted] Title #23095

Open ZhenNan2016 opened 5 days ago

ZhenNan2016 commented 5 days ago

Jira Link: DB-12030

Description

Hi I understand from the following documentation that our ybdb does not support distributed vector storage. May I ask what is the plan for this feature? https://github.com/yugabyte/yugabyte-db/commit/5ca67e496d8c40cdef56c71513ef6d3b6d630596 One of them is mentioned: “Note that the results from a ybdummyann index won’t actually be sorted by their distance from the given query vector as the DocDB side of vector indexing has not been implemented. This is made clear by the following client warning when such an index is created. In the future, when we fully have end-to-end support of vector indexing we will add index AM’s such as hnsw and ivfflat meant for external usage.” Thanks a lot.

Warning: Please confirm that this issue does not contain any sensitive information

mbautin commented 5 days ago

We are working on integrating approximate nearest neighbor search libraries such as usearch and DiskANN, and potentially other libraries, with the distributed storage layer of YugabyteDB. The Postgres interface to this feature will be based on pgvector. The diff you mentioned is adding initial support for vector indexes in the query layer.

tanujnay112 commented 5 days ago

@ZhenNan2016 We will have hnsw and ivfflat as index AMs when we will have built end-to-end support.

ZhenNan2016 commented 5 days ago

We are working on integrating approximate nearest neighbor search libraries such as usearch and DiskANN, and potentially other libraries, with the distributed storage layer of YugabyteDB. The Postgres interface to this feature will be based on pgvector. The diff you mentioned is adding initial support for vector indexes in the query layer.

Very good. Thank you for your reply. May I ask at what point in time exactly will it be supported? Also, will Microsoft spann be supported?

ZhenNan2016 commented 5 days ago

@ZhenNan2016 We will have hnsw and ivfflat as index AMs when we will have built end-to-end support.

Will Diskann and spann be supported?