search

explodinggradients / ragas

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines
https://docs.ragas.io
Apache License 2.0
6.55k stars 643 forks source link

Answer Relevance doesnt need context #1177

Open mohamad-tohidi opened 1 month ago

mohamad-tohidi commented 1 month ago

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug the answer relevance metric, does not need context as input, it doesnt use it anywhere so why is it when i try to run it without contexts list it gives me error?

even in the documentation you mentioned that it needs contexts, but you did not used it anywhere

dosubot[bot] commented 1 month ago

@mohamad-tohidi Hello there! I'm Dosu, a bot here to assist you with bug fixes, questions, and becoming a contributor. I'm here to help while you wait for a human maintainer.

The AnswerRelevance metric does indeed require a contexts list because it uses the contexts to extract relevant sentences necessary to answer the question. The contexts are joined into a single string and then tokenized into sentences for further processing. Without providing a contexts list, the metric cannot perform its intended function, leading to an error.

To resolve the error, ensure that you provide a contexts list when running the metric. The contexts list should be included in the row dictionary with the key "contexts". Here is an example of how to structure the row dictionary:

row = {
    "question": "What is the capital of France?",
    "contexts": ["Paris is the capital of France.", "It is known for its cafe culture and landmarks like the Eiffel Tower."]
}

This will allow the AnswerRelevancy class to properly process the contexts and compute the score [1][2].

The exact error message you receive when the contexts list is not provided is a KeyError, triggered by the following code:

question, contexts = row["question"], row["contexts"]

If row["contexts"] is not provided, it will raise a KeyError [3].

To continue talking to Dosu, mention @dosu.

knpunk commented 1 month ago

@dosu I'd like to add to this that, nowhere in the prompt it's instructing to go through the contexts. It's just taking care of non-committals and for contexts it's just few shot prompting. Maybe due to this I've noticed it's giving better scores for some of the wrong answers as compared to the correct ones. I'm no expert to say this but I think the prompt needs a little more fine tuning.

Cheers and thanks!