search

explodinggradients / ragas

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines
https://docs.ragas.io
Apache License 2.0
6.85k stars 681 forks source link

AttributeError('ContextRecallClassificationAnswers' object has no attribute '__root__') #1450

Open nprime496 opened 3 days ago

nprime496 commented 3 days ago

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug

I cannot evaluate open source gemma:2b model using ragas. I actually get a lot of errors and don't know how to interpret them.

Ragas version: 0.1.21 Python version: 3.10

Code to Reproduce

from datasets import load_dataset
from ragas import evaluate

from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
)
from langchain_ollama.llms import OllamaLLM
from langchain_community.embeddings.fastembed import FastEmbedEmbeddings

# generator with open source models
generator_llm = OllamaLLM(model="gemma2:2b")
# loading the V2 dataset
amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")

embeddings = FastEmbedEmbeddings()

result = evaluate(
    amnesty_qa["eval"],
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall,
    ],
    llm=generator_llm,
    embeddings=embeddings
)

df = result.to_pandas()
df.head()
print(df.head())

Error trace

Evaluating:   1%|▊                                                                | 1/80 [01:48<2:22:32, 108.27s/it]Exception raised in Job[27]: AttributeError('ContextRecallClassificationAnswers' object has no attribute '__root__')
Evaluating:   2%|█▋                                                                | 2/80 [02:15<1:18:28, 60.37s/it]Exception raised in Job[59]: AttributeError('ContextRecallClassificationAnswers' object has no attribute '__root__')
Evaluating:   4%|██▍                                                               | 3/80 [02:57<1:06:54, 52.13s/it]Exception raised in Job[25]: TimeoutError()
Evaluating:   5%|███▍                                                                | 4/80 [03:00<41:14, 32.56s/it]Exception raised in Job[9]: TimeoutError()
Exception raised in Job[60]: TimeoutError()
Exception raised in Job[26]: TimeoutError()
Exception raised in Job[61]: TimeoutError()
Exception raised in Job[62]: TimeoutError()
Exception raised in Job[28]: TimeoutError()
Exception raised in Job[29]: TimeoutError()
Exception raised in Job[8]: TimeoutError()
Exception raised in Job[64]: TimeoutError()
Exception raised in Job[30]: TimeoutError()
Exception raised in Job[1]: TimeoutError()
Exception raised in Job[65]: TimeoutError()
Exception raised in Job[31]: AttributeError('ContextRecallClassificationAnswers' object has no attribute '__root__')
Evaluating:  21%|██████████████▏                                                    | 17/80 [04:06<08:47,  8.38s/it]Exception raised in Job[67]: AttributeError('ContextRecallClassificationAnswers' object has no attribute '__root__')
Evaluating:  22%|███████████████                                                    | 18/80 [04:19<09:09,  8.87s/it]Exception raised in Job[11]: AttributeError('ContextRecallClassificationAnswers' object has no attribute '__root__')
Evaluating:  24%|███████████████▉                                                   | 19/80 [04:51<11:37, 11.44s/it]Exception raised in Job[6]: TimeoutError()
Evaluating:  25%|████████████████▊                                                  | 20/80 [05:15<13:04, 13.07s/it]Exception raised in Job[71]: AttributeError('ContextRecallClassificationAnswers' object has no attribute '__root__')
Evaluating:  26%|█████████████████▌                                                 | 21/80 [05:37<14:21, 14.61s/it]Exception raised in Job[7]: AttributeError('ContextRecallClassificationAnswers' object has no attribute '__root__')
Evaluating:  28%|██████████████████▍                                                | 22/80 [05:55<14:40, 15.19s/it]Exception raised in Job[66]: TimeoutError()
Evaluating:  29%|███████████████████▎                                               | 23/80 [05:57<11:48, 12.43s/it]Exception raised in Job[32]: TimeoutError()
Evaluating:  30%|████████████████████                                               | 24/80 [06:00<09:26, 10.12s/it]Exception raised in Job[33]: TimeoutError()
Exception raised in Job[68]: TimeoutError()
Exception raised in Job[34]: TimeoutError()
Exception raised in Job[69]: TimeoutError()
Exception raised in Job[35]: TimeoutError()
Exception raised in Job[70]: TimeoutError()
Exception raised in Job[36]: TimeoutError()
Exception raised in Job[57]: TimeoutError()
Exception raised in Job[24]: TimeoutError()
Exception raised in Job[58]: TimeoutError()
Evaluating: 100%|███████████████████████████████████████████████████████████████████| 80/80 [15:00<00:00, 11.25s/it]
                                            question  ... context_recall
0  What are the global implications of the USA Su...  ...            NaN
1  Which companies are the main contributors to G...  ...            NaN
2  Which private companies in the Americas are th...  ...            NaN
3  What action did Amnesty International urge its...  ...            NaN
4  What are the recommendations made by Amnesty I...  ...            NaN

[5 rows x 11 columns]

Expected behavior

A result table filled with metrics.

Additional context Add any other context about the problem here.

dosubot[bot] commented 3 days ago

The AttributeError you're encountering with ContextRecallClassificationAnswers is likely due to incorrect usage of the class. The class should be defined with a root attribute, not __root__. Here's how it should be structured:

class ContextRecallClassificationAnswers(RootModel):
    root: t.List[ContextRecallClassificationAnswer]

    def dicts(self) -> t.List[t.Dict]:
        return self.model_dump()

Ensure that your code is using the root attribute correctly. If the error persists, verify that the ContextRecallClassificationAnswers is instantiated properly and that the root attribute is populated with the correct objects [1].

Regarding the TimeoutError, you might want to adjust the timeout settings. The default is 180 seconds, but you can configure it using the RunConfig class:

from ragas.run_config import RunConfig

config = RunConfig(timeout=300)  # Set to a higher value if needed

Pass this configuration to the evaluate() function to see if it resolves the timeout issues [2].

If these steps don't resolve the issue, please provide more specific details about where the error occurs in your code.

To continue talking to Dosu, mention @dosu.