search

explodinggradients / ragas

Supercharge Your LLM Application Evaluations 🚀
https://docs.ragas.io
Apache License 2.0
7.09k stars 719 forks source link

RAGAS for summarization evaluation #814

Closed snassimr closed 5 months ago

snassimr commented 7 months ago

I am pretty newbie in RAGAS

I'd like to evaluate the quality of summarization . I have long context and predicted summarization . How I can use RAGAS for this ?

Up to now I found RAGAS useful only for evaluation of QA applications.

Thanks

shahules786 commented 7 months ago

Hey @snassimr
for now, you could use the faithfulness metric to quantify hallucination in summary. We will add this as an enhancement and add metrics support for the summary. Do you have any recommendations on papers/specific metrics you're looking for?

snassimr commented 7 months ago

Hi , I didn't find some specific paper on some metric . deepeval metric can be some reference https://docs.confident-ai.com/docs/metrics-summarization . This paper Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020). On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661. also can be a reference.

shahules786 commented 7 months ago

Hey @snassimr I'm reopening the issue for keeping track of it. Will mostly add a standalone metric for summarisation by next major release.