search

zylon-ai / private-gpt

Interact with your documents using the power of GPT, 100% privately, no data leaks
https://privategpt.dev
Apache License 2.0
54.22k stars 7.29k forks source link

BGE embedding issue related to Model Dimension #1768

Open nittulukose opened 8 months ago

nittulukose commented 8 months ago

I have managed to deploy privateGPT with sagemaker endpoints. I have used the following models LLM Model : Mistral 7B Instruct Embeding Model: BGE Base En V1.5 (Its dimension is 768) This worked fine.

But when I chang Embedding to any other bge model (dimensions other than 768), I get the following error

ValueError: shapes (4,768) and (384,) not aligned: 768 (dim 1) != 384 (dim 0)

The given error is for the model bge-small-en-v1.5 (Dimension 384).

How can I resolve? I want to use other BGE embedding models with dimensions other than 768.

It seems, while running the privategpt application with "sagemaker mode" is expecting the dimension 768.

How can I explicitly change that to our custom value?

I need to run the embedding model "bge-m3 (dimension 1024)" for multilingual feature.

Looking forward for a solution.

dbzoo commented 8 months ago

You may not mix and match multiple embedding models with different dimensions into the same storage backend without reinitializing them - specifically the vector db. Given that you must re-init the vector store to change dimensions and the index/doc store related to it, you need to reset all database stores.

If you are using simple, chromadb or postgres this will wipe the stores. - See https://github.com/zylon-ai/private-gpt/pull/1772 $ make wipe

nittulukose commented 8 months ago

Hi,

Thank you for your response.

In this case, I am using "qdrant" vectorstore. Could you please let me know how to re-init "qdrant" vector store?

dbzoo commented 8 months ago

If you are using qdrant locally you can delete the directories and files within. Otherwise you need this https://github.com/zylon-ai/private-gpt/pull/1783

nittulukose commented 8 months ago

Hi @dbzoo

Thanks, that worked!!!

I have cleared the contents in the directory, local_data/private_gpt/qdrant. Now, the dimension issue fixed.

Please suggest an LLM - Embedding Model combination that support 'Query Doc' mode for multi language (Both input and output).

My Requirement.

  1. Query Doc Mode
  2. Support multi language (Especially Arabic and English)
  3. Accuracy

Currently I'm using, LLM Model : Mistral 7B Instruct Embeding Model: BGE M3 (Dim 1024) - which claims multilanguage support. But, in this case Query Doc Mode is not accurate.