Data model: Create an individual `embedding` table per `collection`

crate-workbench / langchain

⚡ Building applications with LLMs through composability ⚡

https://python.langchain.com

MIT License

0 stars 0 forks source link

Data model: Create an individual `embedding` table per `collection` #12

Open ckurze opened 7 months ago

ckurze commented 7 months ago

Issue you'd like to raise.

Currently, all embeddings are stored in one single "embeddings" table. This has various downsides:

All embeddings need the same vector length
Indexes get very large
Hard/Complex to manage privileges on database side
Unnecessary join on potentially large amounts of data.

Suggestion:

CrateDB should not create a table "collections" and "emgeddings", but crate a new table that gets the name of the provided collection name in langchain which holds all embeddings.

amotl commented 7 months ago

Hi there,

thank you for writing in, I agree with your proposal.

However, it will need a major rewrite of the data model and adapter, which is currently based on SQLAlchemy, so it may lack some flexibility to do just anything. With your request in mind, another patch has already been submitted to improve the situation in this regard.

GH-13

We will see how the situation can be improved further.

With kind regards, Andreas.

amotl commented 7 months ago

Hi again,

there is a patch now that makes an attempt to improve the situation.

GH-20

With kind regards, Andreas.