dosubot[bot]
commented
1 month ago I found a similar open issue titled Bug in Answer Similarity Calculation, which might be related to your problem. This issue is currently closed and was last updated on June 01, 2024 [1].
Regarding the decrease in the answer_relevancy score, it's important to consider the dependencies and their versions, as they can influence the score. However, I couldn't find specific information on changes in recent versions of the ragas package that could affect the answer_relevancy score [2].
One parameter that could impact the score is the strictness parameter in the ResponseRelevancy class. It determines the number of questions generated per answer, which can affect the score calculation [2].
If you haven't already, you might want to check the versions of the ragas and langchain packages you're using, as changes in these packages could potentially affect the score.
To continue talking to Dosu, mention @dosu.
jinnoh47
[ ] I checked the documentation and related resources and couldn't find an answer to my question.
Your Question The code is identical only difference is the packages version ex) ragas, langchain it answer_relevancy used to be above 0.8 now it is somewhere around 0.7 I have no idea what is the issue.