asg017 / sqlite-vss

A SQLite extension for efficient vector search, based on Faiss!
MIT License
1.59k stars 58 forks source link

Error While Attempting Basic Python Example #97

Closed thewh1teagle closed 10 months ago

thewh1teagle commented 10 months ago

Hello,

I have reviewed the documentation but haven't found a clear way to use it in Python. The task I would like to accomplish is to perform simple insert and search queries.

I attempted the following approach, but unfortunately, the vss_search function didn't work for me.

Code ```python import sqlite_vss import sqlite3 import json # Dummy data. just for example text = "hello world" embeddings = [0.1, 0.2, 0.3] embeddings_to_search = [0.2, 0.3, 0.4] # Initiate sqlite with sqlite-vss conn = sqlite3.connect('db.sqlite') conn.enable_load_extension(True) cur = conn.cursor() sqlite_vss.load(conn) # Create virtual table with text_embedding column which has vectors of size 3 cur.executescript(""" CREATE VIRTUAL TABLE IF NOT EXISTS vss_post USING vss0( embeddings(3) ); CREATE TABLE IF NOT EXISTS post( id INTEGER PRIMARY KEY, text TEXT ); """) # Insert into data table example data and get rowid cur.execute(""" INSERT INTO post(text) VALUES (?) """, (text,)) rowid = cur.lastrowid # Insert actual embeddings, associate it with rowid of data table cur.execute(""" INSERT INTO vss_post( rowid, embeddings ) VALUES (?, ?) """, (rowid, json.dumps(embeddings))) # Search in virtual table to get rowid which we will use to get the data assicated in rowid cur.execute(""" SELECT rowid FROM vss_post WHERE vss_search( embeddings, vss_search_params(?, 1) ) LIMIT 1 """, (json.dumps(embeddings),)) # We should get here the row ID which is similar to vectors_to_search value result = cur.fetchone() print(result) # Close database and cursor cur.close() conn.close() ```
Error ``` terminate called after throwing an instance of 'faiss::FaissException' what(): Error in virtual void faiss::IndexFlat::search(faiss::idx_t, const float*, faiss::idx_t, float*, faiss::idx_t*, const faiss::SearchParameters*) const at /home/runner/work/sqlite-vss/sqlite-vss/vendor/faiss/faiss/IndexFlat.cpp:34: Error: 'k > 0' failed Aborted ```
thewh1teagle commented 10 months ago

I believe the issue is that the virtual table is empty, as I inadvertently omitted the use of db.commit() after inserting both the embeddings and the post text.

edit: solved by using commit(). the error happend because I havn't commited the data, so I performed search over an empty table