Fourth and final branch in the initial series for DiskANN implementation which glue all things together and integrate DiskANN into the libSQL completely 🎉🎉🎉
This PR hooks into the SQLite to make DiskANN indices work:
idxType enumeration extended with new SQLITE_IDXTYPE_VECTOR = 4 (note the if mask 0b11 = 3 will be applied to idxType then vector index type "became" SQLITE_IDXTYPE_APPDEF)
OP_OpenVectorIdx VM op-code were added - although we maybe can change OP_OpenWrite op-code behaviour - new op-code approach is more safe and give us more freedom and safety. For example, P5 register serve it's own need for new op code and if OPFLAG_FORDELETE is set then we will perform truncate of index before allocating new cursor (this is needed for REINDEX command)
OP_IdxInsert and OP_IdxDelete adjusted to handle special vector index type
Disable xferOptimization for vector indices (SQLite optimization which copy internal table/indices structures without explicit reindexing for queries like INSERT INTO t SELECT * FROM q)
Add libsql_vector_idx no-op function which used as a marker in the vector index creation syntax CREATE INDEX t_idx ON t (libsql_vector_idx(e))
Context
Fourth and final branch in the initial series for DiskANN implementation which glue all things together and integrate DiskANN into the libSQL completely 🎉🎉🎉
This PR hooks into the SQLite to make DiskANN indices work:
idxType
enumeration extended with newSQLITE_IDXTYPE_VECTOR = 4
(note the if mask0b11 = 3
will be applied toidxType
then vector index type "became"SQLITE_IDXTYPE_APPDEF
)OP_OpenVectorIdx
VM op-code were added - although we maybe can changeOP_OpenWrite
op-code behaviour - new op-code approach is more safe and give us more freedom and safety. For example,P5
register serve it's own need for new op code and ifOPFLAG_FORDELETE
is set then we will perform truncate of index before allocating new cursor (this is needed forREINDEX
command)OP_IdxInsert
andOP_IdxDelete
adjusted to handle special vector index typexferOptimization
for vector indices (SQLite optimization which copy internal table/indices structures without explicit reindexing for queries likeINSERT INTO t SELECT * FROM q
)libsql_vector_idx
no-op function which used as a marker in the vector index creation syntaxCREATE INDEX t_idx ON t (libsql_vector_idx(e))
vector_top_k(idx, vector, k)
virtual tableTesting
libsql_vector_index.test
TCL based test suitebenchtest
&anntest