In order to enable filtering of documents, we should be able to formulate query trees.
What is a Query Tree
A query tree should be an executable graph on top of our database. If we want to search for the query "insurance draft" only in documents that were created by me, we might create the following query:
In order to do this, we can look up the inverted list for the embeddings of the query and look up the embeddings of author="Matt Barta". Each query operation can be processed in parallel, and since each inverted list is sorted by key, we can guarantee we will scan over document IDs in order.
Per document ,we can determine if it passes the acceptance criteria of matching all terms.
To Consider
I haven't thought out whether this is inefficient from a latency perspective. We aren't far off from a traditional inverted index, so this seems ok.
Secondly, do we have any information available from the embeddings that would lend itself to a "neural WAND" algorithm?
[ ] Create a Query and QueryNode abstraction
[ ] Abstract retrieval to operate within the QueryNode
[ ] Investigate the key structure necessary for arbitrary filter lists to play nice with our current keys.
In order to enable filtering of documents, we should be able to formulate query trees.
What is a Query Tree
A query tree should be an executable graph on top of our database. If we want to search for the query "insurance draft" only in documents that were created by me, we might create the following query:
In order to do this, we can look up the inverted list for the embeddings of the query and look up the embeddings of
author="Matt Barta". Each query operation can be processed in parallel, and since each inverted list is sorted by key, we can guarantee we will scan over document IDs in order.Per document ,we can determine if it passes the acceptance criteria of matching all terms.
To Consider
I haven't thought out whether this is inefficient from a latency perspective. We aren't far off from a traditional inverted index, so this seems ok.
Secondly, do we have any information available from the embeddings that would lend itself to a "neural WAND" algorithm?
see #7