asg017 / sqlite-vss

A SQLite extension for efficient vector search, based on Faiss!
MIT License
1.59k stars 58 forks source link

"Error reconstructing vector" when reading a newly inserted vector from different processes #120

Open sigmike opened 4 months ago

sigmike commented 4 months ago

When a process adds a row to a vss table, any other process that has already connected to the same database will get the following error when accessing the newly added vector:

sqlite3.OperationalError: Error reconstructing vector - Does the column factory string end with IDMap2? Full error: Error in void faiss::IndexIDMap2Template::reconstruct(faiss::idx_t, typename IndexT::component_t*) const [with IndexT = faiss::Index; faiss::idx_t = long int; typename IndexT::component_t = float] at /home/runner/work/sqlite-vss/sqlite-vss/vendor/faiss/faiss/IndexIDMap.cpp:236: key 0 not found

(The key is the rowid of the added row)

Restarting the reading process resolves the error.

The solution may be the same as for #64.

This looks difficult to fix, so maybe just add a note about this limitation in the README for now?