Closed amaiya closed 11 months ago
🤖
Hello,
Thank you for bringing this to our attention. The issue you're experiencing seems to be related to the way similarity scores are calculated in the Chroma class of LangChain.
In the Chroma class, the similarity_search_with_score
method is used to calculate similarity scores. This method returns a list of documents most similar to the query text along with their cosine distance scores. Lower scores represent more similarity.
However, the scores you're seeing are negative, which is not expected. The scores should be between 0 and 1, where 0 is completely dissimilar and 1 is most similar.
This could be due to a bug in the similarity_search_with_relevance_scores
method in the VectorStore class, which is used when the search type is set to "similarity_score_threshold". This method is supposed to filter out any documents with a similarity score less than the score_threshold. If no documents meet the score_threshold, a warning is issued.
I would recommend checking the version of the chromadb
library you're using, as this could potentially be causing the issue. If you're using an older version, try updating to the latest version and see if the issue persists.
If the issue still persists after updating chromadb
, it would be helpful if you could provide more information about the documents you're using for testing, as well as the exact version of langchain
and chromadb
you're using. This will help us reproduce the issue and find a solution.
Thank you for your patience and cooperation.
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
For anyone else who stumbles on this, the missing piece appears to be collection_metadata={"hnsw:space": "cosine"}
. Without it, negative scores are returned in the current version of chromadb and langchain. If you supply client
or client_settings
to Chroma.from_docments
, you also may need to explicitly supply a collection_name
other than langchain
for some reason.
This GitHub repo is helpful.
For anyone else who stumbles on this, the missing piece appears to be
collection_metadata={"hnsw:space": "cosine"}
. Without it, negative scores are returned in the current version of chromadb and langchain. If you supplyclient
orclient_settings
toChroma.from_docments
, you also may need to explicitly supply acollection_name
other thanlangchain
for some reason.This GitHub repo is helpful.
I highly appreciate if you could let me know to set collection_metadata and in what function call for langchain with aws bedrock
I highly appreciate if you could let me know to set collection_metadata and in what function call for langchain with aws bedrock
You supply collection_metadata
to Chroma
when instantiating langchain.vectorstores.Chroma
retriever_score_threshold = vector_db.as_retriever( search_type="similarity_score_threshold", search_kwargs={'score_threshold': 0.1,'k':5} ) retrieved_docs = retriever_score_threshold.invoke("xxx") retrieved_docs ,in langchain 0.2 it doesn't work OMG ,still get negative socre
I encountered the same issue as described. Initially, I was uncertain about the root cause, so I experimented with different configurations. Here’s a brief overview of what I tried and what eventually worked for me:
Initially, I used the text-embedding-3-small
model with the following setup:
persist_directory = get_absolute_path('/app/chroma/')
persistent_client = chromadb.PersistentClient(path=persist_directory)
vector_db = Chroma(
client=persistent_client,
persist_directory=persist_directory,
collection_name="collection-foobar",
embedding_function=OpenAIEmbeddings(),
collection_metadata={"hnsw:space": "cosine"}
)
retriever = vector_db.as_retriever(
search_type="similarity_score_threshold",
search_kwargs={'score_threshold': 0.6},
k=10
)
With this configuration, I encountered issues with negative scores. After switching to the text-embedding-ada-002
model, the problem was resolved. If anyone else tries this solution, please confirm whether it works for you as well. Your feedback would be greatly appreciated!
I am facing the same issue:
vectorstore = Chroma(collection_name=collection_name, embedding_function=embeddings_model, client=chroma_client.client, collection_metadata={"hnsw:space": "cosine"})
I am using the similarity_search_with_relevance_scores
this is a part of the sample output
[(), -219.28074010897063)] warnings.warn(UserWarning: No relevant docs were retrieved using the relevance score threshold 0.7)
I did run into the same negative similarity scores using local Ollama models.
@matardy Thanks for experimenting with models and configs. But shouldn't negative scores never occur whatever model is used? Maybe it's worthwhile to re-open the issue.
@matardy, literally the same issue when moving to the new 3-series embedding models from openai, even encounter this when changing to max inner product. Did you figure out why this is happening?
@josefansinger how did you resolve the issue?
I did not. Sorry. Also in the LangChain Discord no response at all.
Should we reopen this ?
Just tested with the new gpt embedding model "text-embedding-3-large" same issue
Hi, any other free opensource alternative to text-embedding-3-small
. Please let me know.
I used vector_store = FAISS.from_documents(text_chunks, embedding=embeddings, normalize_L2=True)
and used huggingface embedding to get similarity score between 0 to 1.
Solution : normalize_L2=True
Nope, that didn't work!, Plus that's for FAISS the problem is on Chroma DB
If you're looking for a free open-source alternative to the text-embedding-3-small model, mxbai-embed-large is an excellent choice!
I was now able to recreate this consistently. have a script, happy to share, the issue is from the FAISS wrapper. The openai wrapper works correctly but somewhere in FAISS there are critical inconsistencies. Will try to debug mode but this is a thing.
Furthermore, querying the same faiss index with text vs with vector yields different results which makes matter even worse. (I embed it separately to do the later)
I am having same issue. When I use HuggingFace embedding model Alibaba-NLP/gte-large-en-v1.5
, it shows this problem with the FAISS vector db. When I try same code with Open AI text-embedding-3-large
embedding model this doesn't happen. It gives score between 0 and 1. As suggsted above, the normalize_L2
argument makes the scores even worse.
I was now able to recreate this consistently. have a script, happy to share, the issue is from the FAISS wrapper. The openai wrapper works correctly but somewhere in FAISS there are critical inconsistencies. Will try to debug mode but this is a thing.
Furthermore, querying the same faiss index with text vs with vector yields different results which makes matter even worse. (I embed it separately to do the later)
Hi, it would help to share the script. I'm curious to take a look :)
Hey all, I tried to research the bug, using this time the latest modules and the initial script :
langchain==0.2.7
chromadb==0.5.3
And the initial script ending in:
print(retriever.get_relevant_documents('Ketanji'))
I didn't get any scores at all, just an empty list (with the initial _scorethreshold of 0.2):
The omission of scores is caused by this line:
But I still don't see any negative score, even though they are (expectedly) low:
I tried this with llama3 and mistral and get negative scores
So I found out that the issue/feature is with the search with relevance score method that vector engines inherit from base VectorStore. The relevance method applies an additional relevance function which causes the negative similarities. This is an issue if your vectorstore is already returning valid distances (most cases).
All we had to do is use the other search methods like search with scores since our vector database already provides valid distances.
@pedrojrv thanks for the hint, if that is the case how does one pass the threshold to db.similarity_search_with_score(query=query, k=3)
?
Getting negative scores with this model: gtemicro = SentenceTransformer('Mihaiii/gte-micro-v3')
. The collection_metadata arg did not solve it for me...
Version: 0.2.11
System Info
langhcain==0.0.297
chromadb==0.4.12
Who can help?
@hwchase17 @agola11
Information
Related Components
Reproduction
The above code returns negative similarity scores for all retrieved results:
Expected behavior
Score for each
Document
should be between 0 and 1.