search

explodinggradients / ragas

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines
https://docs.ragas.io
Apache License 2.0
6.78k stars 667 forks source link

[R-275] Tokens Usage for evaluations and testset generations #540

Closed aliai22 closed 2 months ago

aliai22 commented 8 months ago

I am using RAGAs evaluation metrics to evaluate my RAG pipeline. Is there any way I could know the openai tokens used per api call from RAGAs, lets say for any one metric, i.e., faithfulness. I want to estimate the cost of using RAGAs evaluators.

R-275

jjmachan commented 8 months ago

this is a valuable suggestion @aliai22 :)

we have thought about it and will be implementing it soon but it might take a couple of weeks to prioritize this. The reason is that I want to take some time to design how stats would be like

basically, we want to track things like

in a unified way. I'll explore more and send an RFC your way asking for your feedback as soon as I have it ready. would that help?

aliai22 commented 7 months ago

Yeah, that would be great. Looking forward to hear from you.

Vishnu-add commented 5 months ago

Did it get implemented?

fhuthmacher commented 4 months ago

I am also looking for this, any update?

abetatos commented 4 months ago

Pinging to see if this gets released at any point.

nfsergiu commented 3 months ago

Any updates on this @jjmachan ?

whitney-meer commented 3 months ago

I'm also interested in this!

jjmachan commented 3 months ago

hey folks, not yet but I will take this up this week and have something ready for your feedback shortly, my sincere apologies for the delay!

jjmachan commented 2 months ago

fixed in #1129