search

stanford-futuredata / ARES

https://ares-ai.vercel.app/
Apache License 2.0
386 stars 43 forks source link

RAGAS score calculation from annotations is unclear #4

Closed pchunduri6 closed 7 months ago

pchunduri6 commented 7 months ago

I'm not sure how the RAGAS score is computed from annotations in RAG_Automatic_Evaluation/RAGAS_Scoring.py:

# Lines 68-72
sampled_y_labels = dataset.sample(n=300, random_state=42)
context_relevance_prediction = sum(dataset["Context_Relevance_Label"].tolist()) / len(sampled_y_labels)
answer_relevance_prediction = sum(dataset["Answer_Relevance_Label"].tolist()) / len(sampled_y_labels)
context_scores.append(context_relevance_prediction)answer_relevance_scores.append(answer_relevance_prediction)

While I'm not sure what this code is trying to compute, I ran it to sanity check, and I got nan outputs:

image

Any help understanding this issue and pointers to the relevant sections in the paper would be greatly appreciated. Thanks!

jonsaadfalcon commented 7 months ago

I believe that was a bug with an earlier version. It should work fine now. Thanks!