A user should be able to filter documents - Githubissues

DeployQL / LintDB

Vector Database with support for late interaction and token level embeddings.

https://www.lintdb.com/

Apache License 2.0

40 stars 0 forks source link

A user should be able to filter documents #7

Open mtbarta opened 2 months ago

mtbarta commented 2 months ago

As a user of LintDB, I want to be able to filter by metadata in a search so that I can more accurately retrieve my data.

Acceptance Criteria

We can filter on a. strings b. dates
Metadata can be returned in the search response.

We need to consider how data will be stored on disk.

Flatbuffers needs to precompile the data structures, which prevents users from specifying a schema. Should a user specify a schema to store metadata?

We also need to consider how we'll filter data without duplicated the metadata.

My first thought is that we need to create inverted lists for the attributes and create a retrieval process that looks like document-at-a-time. We might want a query engine to accomplish this.

Notes

I'm curious who has a good version of a query engine to think about. Quickwit is awesome and might have something similar?