Feature request: Support a vector database

agnaistic / agnai

AI Agnostic (Multi-user and Multi-bot) Chat with Fictional Characters. Designed with scale in mind.

https://agnai.chat

GNU Affero General Public License v3.0

535 stars 107 forks source link

Feature request: Support a vector database #349

Open BasadoBased opened 1 year ago

BasadoBased commented 1 year ago

Supporting a vector database like ChromaDB would have numerous benefits including a longer memory.

sceuick commented 1 year ago

There are plans to start bringing in features like this. If you have any expertise with ChromaDB, your input or what the implementation would like would greatly be appreciated over in the Discord or here.

BasadoBased commented 1 year ago

Unfortunately I don't however there has been some (unfinished) work on this recently in Tavern that might be a helpful reference: https://github.com/Cohee1207/SillyTavern/pull/333 https://github.com/Cohee1207/SillyTavern-extras/issues/23 and oobabooga implementation: https://github.com/oobabooga/text-generation-webui/tree/main/extensions/superbooga

rondlite commented 1 year ago

I used weaviate:

https://github.com/weaviate/weaviate?utm_source=google&utm_medium=cpc&utm_campaign=17432327206&utm_content=142705996452&utm_term=weaviate&gclid=CjwKCAjwpayjBhAnEiwA-7ena-EGwc_xzO26FW7fjN8R2hcIMxXouwpeIvQSKWonxtRk-5t9XUEB-RoCs5AQAvD_BwE And Pinecone (free account):
https://www.pinecone.io/

Chromadb seems to be focussed towards certain LLMs and is embeddings.

I think ChromaDB will give the best results if used correct but will be the hardest to use in a 'one size fits all' scenario, since it runs a selected LLM index

sceuick commented 1 year ago

Those two look promising, however pinecone appears to be a PaaS and weaviate needs to be installed. Weaviate definitely looks promising, but I need a low barrier for entry which ChromaDB satisfies by being installable via pip/poetry.

rondlite commented 1 year ago

Weaviate client is installed with pip install weaviate-client or even better npm i weaviate-ts-client

Keep in mind that the server part in all cases uses very large amounts of RAM/VRAM as in A100-80GB (recommended in this list https://www.e2enetworks.com/blog/top-7-vector-databases-for-ai). I am running weaviate and it takes up 32GB of memory on a dev server, this is why I recommended Pinecone because of the large free tier and most automation gpt packages (autogpt,babyagi) support it for that reason.

Another vector db to not overlook is Milvus

I tried all of the above and weaviate for me was quickest to implement due to a docker for the server and npm package for the client, zero config.