Open asg017 opened 1 year ago
Use IDSelectorBatch
.
need to figure out idxStr/idxNum rules
I suppose then we can do pre-filtering with standard SQL and then feed the resulting rowids into the vss query. Nice.
Definitely nicer than getting 1000 "wide net" results from the vss query and then filtering.
That being said, in a document searching app I am working on I do 50 top searches and then filter, seems to work, albeit we never have the certainty that we are not missing something important.
Use
IDSelectorBatch
.need to figure out idxStr/idxNum rules
I didn´t get the meaning of this one - can you please expand for us noobies?
Those are mostly personal notes about how to implement this feature. IDSelectorBatch is Faiss tool that'll make it easier to search a large subset of vectors, and idxStr/idxNum refer to some internal changes I need to make to the vss0
module in order to make this compatible with older code.
I'll probably work on this next after the new v0.1.1
releases this week!
Any update on this? @asg017
In KNN style searches, we should support
rowid in (...)
constraints in queries like so:Currently we ignore the "equals" constraint on
rowid
, but if we were to capture that constraint (and enablesqlite3_vtab_in
), we could read in all the rowids and use IDSelector to pre-filter results.This would be especially great when paired with subqueries:
This would enable "pre-filtering" according to this post. This would be an easy-to-implement but probably-slow solution to push-down filters described in #2.