tortoise / tortoise-orm

Familiar asyncio ORM for python, built with relations in mind
https://tortoise.github.io
Apache License 2.0
4.58k stars 378 forks source link

Add vector field (pgvector, sqlite-vss, etc) #1467

Open veqtor opened 1 year ago

veqtor commented 1 year ago

TL;DR make it possible to use tortoise-orm as a vector database

Is your feature request related to a problem? Please describe. A lot of commercial vector databases are being thrown around but none of them adequately support your own schemas and never have the ability to use an ORM, pgsql+pgvector is probably the best choice so it would be good to support it in tortoise-orm .

Describe the solution you'd like Add a new vector field, if using a postgresql database, it uses pgvector and if using sqlite it uses sqlite-vss. Ideally it would allow adding/updating an index after rows have been added (recreate possible?)

Describe alternatives you've considered The only real alternative is to not use tortoise-orm

Additional context Vectors are used to store embeddings, such as those coming from OpenAI, they can be used to retrieve documents based on their semantic similarity, images by content and so on.

Links: https://towardsdatascience.com/explaining-vector-databases-in-3-levels-of-difficulty-fc392e48ab78 https://platform.openai.com/docs/guides/embeddings https://github.com/pgvector/pgvector https://github.com/asg017/sqlite-vss