feast-dev / feast

The Open Source Feature Store for Machine Learning
https://feast.dev
Apache License 2.0
5.55k stars 993 forks source link

Add support for Python 3.11 and Ubuntu for SQLite #4263

Open franciscojavierarceo opened 4 months ago

franciscojavierarceo commented 4 months ago

Is your feature request related to a problem? Please describe. Can't run sqlite_vec on 3.11 or non-Darwin OS.

Describe the solution you'd like Able to run sqlite vector search on Linux and not limited to 3.11

Describe alternatives you've considered N/A

Additional context https://github.com/feast-dev/feast/issues/4209

asg017 commented 4 months ago

@franciscojavierarceo I think this is because there's a SQLite bug on version < 3.41 on LIMIT ? clauses on virtual tables. In a lot of sqlite-vec docs the example KNN query is:

select 
  rowid, 
  distance
from vec_example
where column match ?
order by distance
limit 100

But this query won't work on SQLite version < 3.41, which is causing this bug.

The workaround is to provide the limit as a WHERE clause like so:

select 
  rowid, 
  distance
from vec_example
where column match ?
  and k = 100
order by distance

I'll be sure to document this more - tbh I might not even show the LIMIT 100 variant anymore, since many people are stuck on old SQLite versions in practice

franciscojavierarceo commented 4 months ago

Thanks!

FYI, I just added support for sqlite_vec in Feast here: https://github.com/feast-dev/feast/pull/4176

Thank you for this package!