dosubot[bot]
commented
1 year ago Answer generated by a š¤
Answer
Hello,
Thank you for bringing this to our attention. The behavior you're observing is likely due to the nature of the RAG (Retrieval-Augmented Generation) model used in the LangChain framework.
RAG models are designed to retrieve relevant documents from a database and use them to generate responses. When the database is sparse or contains only a few documents, the model might still try to generate a response based on the limited information it has. This could explain why you're seeing an answer for a query about "Sachin Tendulkar" even though your database only contains a football article.
As for the reference link, it's possible that the model is defaulting to a pre-set link when it can't find a relevant document in the database. This could be a fallback mechanism to ensure that a link is always provided, even if it's not directly relevant to the query.
When you add more articles to the database, the model has more information to work with and can make more accurate retrievals. This could explain why you're not seeing an answer for the "Sachin Tendulkar" query when your database is more populated.
As for the expected behavior, it's indeed reasonable to expect that the model would not provide an answer if the query is not mentioned in the database. However, RAG models are designed to generate responses even when the exact query is not present in the database. They do this by leveraging the semantic similarity between the query and the documents in the database.
To better control the behavior of the model, you might want to adjust the retrieval and generation parameters. For example, you could increase the retrieval threshold to ensure that only highly relevant documents are retrieved. You could also adjust the generation parameters to control the length and diversity of the generated responses.
I hope this helps clarify the behavior you're observing. If you have any further questions, please don't hesitate to ask.
Best regards, Dosu
Sources
- libs/langchain/langchain/chains/sql_database/prompt.py
- libs/langchain/langchain/memory/chat_message_histories/rocksetdb.py
This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a š if this is helpful and š if it is irrelevant.
System Info
Ubuntu 22 Langchain version
Who can help?
No response
Information
Related Components
Reproduction
Current behaviour:
I'm using the RAG example and feeding my own database of 1 football article. The Pinecone DB is a brand new database and only contains vectors from the football article.
When I do qa_with_sources(query="Who is Sachin Tendulkar") it provides me an answer and a link as a reference. This is not the expected behavior.
I have not fed any article about Sachin Tendulkar to the database. How and why/where from is it getting the answer and the link?
Now, If I add more articles only about football, push the vector count in the database to around 90. And then I ask the same question, query="Who is Sachin Tendulkar", it is not able to give the answer, which is the expected behavior.
I wonder if the fullness of the vector db makes it more accurate? Has anyone else seen this?
Repro:
Create a new Vector DB on pinecone. Use this example to feed in a football article.
Run query="Who is Sachin Tendulkar". Note the result contains a reference and an answer. (Unexpected)
Now, create a more full db, with more articles and ask the same query. Note that the results is empty as expected.
Expected behavior
Since the database does not contain any article or mention of Sachin Tendulkar, it should not provide any answer, and instead say "This is not mentioned in the database".