FAISS - Incorrect warning and relevance score when using `MAX_INNER_PRODUCT` and `normalize_L2`

[zhichenggeng]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from langchain_community.vectorstores import FAISS
from langchain_aws import BedrockEmbeddings

texts = ["I like apples", "I like oranges"]
distance_strategy = "MAX_INNER_PRODUCT"

embeddings = BedrockEmbeddings(
    region_name="us-east-1", model_id="amazon.titan-embed-text-v1"
)

normalize_L2 = False
vectorstore = FAISS.from_texts(
    texts,
    embeddings,
    distance_strategy=distance_strategy,
    normalize_L2=normalize_L2
)
results = vectorstore.similarity_search_with_score("I like apples", k=1)
print(results)

normalize_L2 = True
vectorstore = FAISS.from_texts(
    texts,
    embeddings,
    distance_strategy=distance_strategy,
    normalize_L2=normalize_L2
)
results = vectorstore.similarity_search_with_score("I like apples", k=1)
print(results)
results = vectorstore.similarity_search_with_relevance_scores("I like apples", k=1)
print(results)

Error Message and Stack Trace (if applicable)

No response

Description

According to FAISS, the cosine similarity can be obtained by normalizing the vectors first and then using inner product to build the index. However, running the code above, when setting normalize_L2 = True, there will be a warning:

UserWarning: Normalizing L2 is not applicable for metric type: MAX_INNER_PRODUCT

Moreover, the relevance score is counterintuitive. If we are computing cosine similarity, the relevance score from similarity_search_with_relevance_scores should be identity to the score from similarity_search_with_score. However, the implementation will lead to smaller relevance score when two vectors are closer. https://github.com/langchain-ai/langchain/blob/fd546196ef0fafa4a4cd7bb7ebb1771ef599f372/libs/core/langchain_core/vectorstores/base.py#L422-L427

System Info

System Information

OS: Linux OS Version: #1 SMP Tue May 21 16:52:24 UTC 2024 Python Version: 3.10.13 | packaged by conda-forge | (main, Oct 26 2023, 18:07:37) [GCC 12.3.0]

Package Information

langchain_core: 0.2.10 langchain: 0.2.6 langchain_community: 0.2.6 langsmith: 0.1.82 langchain_aws: 0.1.8 langchain_text_splitters: 0.2.2 langgraph: 0.1.3

[zhichenggeng]

There are some issues mentioned about the relevance score problem previously: https://github.com/langchain-ai/langchain/issues/9519, https://github.com/langchain-ai/langchain/issues/22209, and https://github.com/langchain-ai/langchain/issues/14948. The normalize_L2 problem is never discussed. I guess that's because many embedding models normalize the embeddings, while titan embedding model doesn't do that.

Running the above code will give the following output:

[(Document(page_content='I like apples'), 388.789)]
[(Document(page_content='I like apples'), 1.0000002)]
[(Document(page_content='I like apples'), -2.384185791015625e-07)]

[zhichenggeng]

If this behavior is truly unexpected, I could help submit a PR to solve this.