Query about calculation of metrics

[anubhav562]

Hey,

Thanks for creating and maintaining this repository.

I assume that you would be using an LLM to get out the scores for each metric. Or are you guys using some bespoke model for each metric like coherence, faithfulness?

If you rely on an LLM, how do you get the score? Do you ask the LLM to spit out a score?

Please let me know!

Thanks!

[shahules786]

Hi, @anubhav562 , thanks for asking. We do not use LLM directly to score the model since it can be biased instead we design paradigms that leverage LLMs to give certain outputs and then use algorithms to quantify the output. You can observe that all of ragas scores are between 0 and 1.
If you want to read more about metrics refer to our docs.