Open amitjoy opened 1 month ago
To integrate EvaluatorChain
with your existing Runnable RAG chain, you need to instantiate the EvaluatorChain
with the appropriate metric and then use it within your RAG chain. Below is an example of how you can achieve this:
Instantiate the EvaluatorChain
:
from ragas.metrics import SomeSpecificMetric # Replace with the actual metric you want to use
from ragas.langchain.evalchain import RagasEvaluatorChain
# Initialize the metric
metric = SomeSpecificMetric()
# Create the EvaluatorChain
evaluator_chain = RagasEvaluatorChain(metric=metric)
Integrate the EvaluatorChain
into your RAG chain:
Assuming you have an existing RAG chain, you can call the EvaluatorChain
within it. Here is a simplified example based on your provided code:
from langchain.chains import SomeRAGChain # Replace with your actual RAG chain
def chain(self) -> Runnable:
"""
Constructs the retrieval chain for the chat agent.
:return: A Runnable instance representing the retrieval chain.
"""
logger.debug("Initializing retrieval from knowledge base chain")
history_aware_retriever = create_history_aware_retriever(
llm=self.vertex.model,
retriever=self.kb_agent.retriever,
prompt=self.condense_prompt
)
document_chain = create_stuff_documents_chain(
llm=self.vertex.model,
prompt=self.prompt
)
retrieval_chain = create_retrieval_chain(
retriever=history_aware_retriever,
combine_docs_chain=document_chain
)
def get_session_history(session_id: str) -> BaseChatMessageHistory:
return self.history_agent.message_history
# Integrate the EvaluatorChain
def evaluate_chain(inputs):
rag_outputs = retrieval_chain(inputs)
evaluation_result = evaluator_chain(rag_outputs)
return evaluation_result
return RunnableWithMessageHistory(
evaluate_chain,
get_session_history,
input_messages_key="input",
history_messages_key="chat_history",
output_messages_key="answer"
)
In this example, replace SomeSpecificMetric
with the actual metric you want to use and SomeRAGChain
with your actual RAG chain class. The evaluate_chain
function runs the RAG chain and then evaluates the outputs using the EvaluatorChain
.
This integration allows you to evaluate the outputs of your RAG chain using the specified metric within the EvaluatorChain
[1][2].
To continue talking to Dosu, mention @dosu.
@amitjoy where you able to fix this?
[ ] I checked the documentation and related resources and couldn't find an answer to my question.
Your Question How can I integrate EvaluatorChain with an existing Runnable RAG chain?
Code Examples An example of chain created using LCEL:
Additional context running on latest versions of Langchain and Ragas