Open tomusher opened 6 months ago
Similar to Wagtail and Django, it would probably be good to let users to find their own indexes. That way they can choose the constraints and embedding model for example they might not want to use OpenAI or might use variations of OpenAI models that have various embedding sizes. Embedding choice should be a project decision. Instead of using a default index size it might be good to provide documentation and helpers for creating indexes.
The
PgvectorEmbedding
model doesn't currently includepgvector
indexes. Adding these would speed up retrieval for large datasets.Adding an index is relatively straightforward with the
pgvector
library, but the problem is that we need to specify the number of dimensions on theVectorField
.We currently leave this undefined because the user could specify multiple embedding backends with different output dimensions, which we'd be keeping in the same table.
In order to specify dimensions and therefore use indexes, we'd need to:
Leaning towards only supporting one embedding model - I expect this is going to be OK especially with the increasing prevalence of multimodal embedding models through which we could embed images, audio, pages, etc. with the same model.