tursodatabase / libsql

libSQL is a fork of SQLite that is both Open Source, and Open Contributions.
https://turso.tech/libsql
MIT License
9.54k stars 252 forks source link

vector search: hide vector index for sqlite planner #1678

Closed sivukhin closed 1 month ago

sivukhin commented 1 month ago

Context

PR https://github.com/tursodatabase/libsql/pull/1594 tried to solve issue with sqlite planner using vector index for some optimizations (for example in SELECT COUNT(*) FROM t queries).

Turned out hacks to avoid usage of vector indices in planner were not enough. Actually, bUnordered field set by sqlite3 during index analysis process and manipulating with it can be dangerous and error-prone (as sqlite3 can reset it at any moment).

So, this PR fixes this more robustly:

  1. opMask is set to 0 for vector index in whereLoopAddBtreeIndex which must additionally prevent sqlite to use index in any operations (>, >=, =, IN) (actually, it shouldn't happen because index has expression - but still - this is another guard)
  2. whereIsCoveringIndex return false for vector index
  3. Loop over indices in query planner skips vector indices completely
  4. All places where optimizer prevented to use bUnordered indices were extended to have additional check for idxIsVector field