dosubot[bot]
commented
5 months ago To resolve the issue with context selection in a llama_index RAG application when chat_mode is set to "context", you should use the select_context method provided by the TruChain class. This method is designed to get the path to the context in the query output.
Here is an example of how you can use the select_context method:
import numpy as np
from llama_index.core import VectorStoreIndex
from llama_index.readers.web import SimpleWebPageReader
from trulens_eval import TruLlama, Feedback, Tru, feedback
from trulens_eval.feedback import Groundedness
tru = Tru()
tru.reset_database()
def evaluate(app_id, chat_engine, tru):
openai = feedback.OpenAI()
# Correct context selection using select_context
context_selection = TruLlama.select_context(chat_engine)
# Question/answer relevance between overall question and answer.
f_qa_relevance = Feedback(openai.relevance, name="QA Relevance").on_input_output()
grounded = Groundedness(groundedness_provider=openai)
f_groundedness = (
Feedback(grounded.groundedness_measure_with_cot_reasons, name="Groundedness")
.on(context_selection)
.on_output()
)
f_context_relevance = (
Feedback(
openai.qs_relevance_with_cot_reasons,
np.max,
name="Context relevance",
)
.on_input()
.on(context_selection)
)
tru_recorder = TruLlama(
chat_engine,
app_id=app_id,
tru=tru,
feedbacks=[f_context_relevance, f_groundedness, f_qa_relevance],
)
question = "What did the author do growing up?"
with tru_recorder as recording:
response = chat_engine.chat(question)
print(response)
app_id = "test_chat_engine"
documents = SimpleWebPageReader(html_to_text=True).load_data(
["http://paulgraham.com/worked.html"]
)
index = VectorStoreIndex.from_documents(documents)
chat_engine = index.as_chat_engine(chat_mode="context")
evaluate(app_id, chat_engine, tru)
tru.run_dashboard()This code snippet demonstrates how to select the context using the select_context method and use it in a feedback function [1][2].
To continue talking to Dosu, mention @dosu.
piotrm0
mlevtov
sfc-gh-pmardziel
sfc-gh-jreini
Bug Description When evaluating a llama_index rag application, context selection does not work properly when chat_mode="context"
To Reproduce
Expected behavior This should run and provide context relevance, answer relevance, and groundedness. This is what happens if you remove
chat_mode="context"Relevant Logs/Tracebacks
Environment:
Additional context Should I be setting
context_selectionto something else?