The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.
llm_context_precision: the metric considers the context used to generate the question-answer pair in step 2 (02_qa_generation.py), instead of the retrieved contexts. To assess the ability of the system to retrieve relevant chunks/contexts, we need to consider the relevancy of the retrieved contexts against the question. The computation of the metric is also using as input the generated answer (actual) instead of the question. The updated metric computes a simple average precision (proportion of relevant chunks without consideration of the ranking order).
llm_context_recall: this metric is also using the qna context instead of the retrieved contexts. To assess the ability of the system in retrieving contexts that are aligned with the ground truth answer we need to consider the retrieved contexts. The temperature parameter is removed as it's already part of the initialisation of ResponseGenerator.
llm_answer_relevance: the metric currently inputs the generated answer and the ground truth answer instead of the question and the generated answer.
This PR fixes:
02_qa_generation.py), instead of the retrieved contexts. To assess the ability of the system to retrieve relevant chunks/contexts, we need to consider the relevancy of the retrieved contexts against the question. The computation of the metric is also using as input the generated answer (actual) instead of the question. The updated metric computes a simple average precision (proportion of relevant chunks without consideration of the ranking order).ResponseGenerator.