dssjon / biblos

www.biblos.app
http://www.biblos.app
Other
197 stars 14 forks source link

Testament Filter #2 #9

Closed HanClinto closed 11 months ago

HanClinto commented 11 months ago

Feature Summary

NOTE: This implementation does NOT require any additional metadata to be added to the database. However, during setup it does need to iterate through all of the data to compile a list of books of the Bible. This operation does add some latency to startup (because it queries every item in the index), but because everything is local, it's hopefully not too bad.

NOTE: This implementation is also less than ideal in that it creates a very large OR clause that joins all books of the Old and New Testaments together to filter metadata. I.E.,

filter = {"$or": [{"book": "GEN"}, {"book": "EXO"}, {"book": "LEV"} ... ]}

It can be a bit of a long conditional clause, but it still seems to be pretty zippy running locally on my Macbook Pro.

And on the plus side, no database schema updates are required in order for this to function! :)

Feature Screenshot

image
HanClinto commented 11 months ago

Huzzah, was able to figure out the syntax for how to do a large $or query (never would have figured this out without the help of this example: https://github.com/chroma-core/chroma/blob/main/examples/basic_functionality/where_filtering.ipynb

So this doesn't require the addition of any metadata in the database, which is very nice. As a downside, it's a large $or filter which does slow down the query time (as can be seen by the performance timer), but it's still pretty zippy (as can also be seen by the performance timer) so the difference is pretty negligible. :)

HanClinto commented 11 months ago

I see the conflicts -- will rebase to resolve. Uno momento...

dssjon commented 11 months ago

@HanClinto deployed :)

HanClinto commented 11 months ago

@HanClinto deployed :)

Looks great! Really satisfying to see this in production, thanks! :D