Closed Saurabh8255 closed 3 weeks ago
It seems like you have a question about the Ragas library. Could you please provide more details about what you're unclear about or what specific information you're looking for? If you have any code snippets or additional context, that would also be helpful in understanding your question better.
To continue talking to Dosu, mention @dosu.
@Saurabh8255 I was wondering about a similar issue, the usefulness of the critic
in the TestsetGenerator.from_langchain
method (I infer you were talking about this method since there is no critic
parameter in the evaluate
method) and after having looked for it in the repository, I can't find any reference to the critic
parameter apart from the definition of the TestsetGenerator.critic_llm
attribute.
I'll let @jjmachan or @shahules786 bring a clear answer and conclude this issue, but I think it might be a relic of a previous version.
After some research I found that it will be better if we use different LLM as critic but make sure that it will me larger model than we used in our RAG architecture. for better evaluaion.
Yeah, you are theorically right but also empirically right ! I couldn't find where critic_llm
was used but I have finally managed to find it : it is used in TestsetGenerator.init_evolution:l226-232
to define filters as described in the guide Using Ragas Critic Model instead of GPT-4.
I see you folks have figure out the answer 🙂 but I'll add my thoughts here too. The reason why there is both generator and critic models is that we found out when developing that for the critic parts of the pipeline (which decides if the generated question or the chosen contexts if useful for further processing) needs a better model for more sterability so we have more control over specifing what the correct parameters are.
We did a small experiment to make a custom model for this which @Gwenn-LR has shared. but we are revamping the testset generation tools with #1016
Thanks for your answer @jjmachan but also all the amazing work of your team !
I'm eager to read your opion about my analysis and to see the evolution of this package. Do not hesitate to ask for any help if needed, I would be happy to lend you a helping hand !
Have a nice day !
@Gwenn-LR thank you for the writeup but it was a bit long so I didn't quite understand what you actually had in mind so correct me if I'm wrong
but for this
embeddings =? rag_embedding critic_llm = llm >= generator_llm > rag_llm
the correct order is embeddings = rag_embeddings (right now since we don't see a lot of diff in perf and we don't have a big conponent of retriever) it maybe said that embedding < rag_embedding because the actual usecase for retriever is not that much
2nd one is roughly correct but only because of the current setup. generator_llm = rag_llm though. critic_llm can be a smaller fine-tuned model
does address all of your thoughts?
Thanks for your answer and yes it clearly address my point!
It seems the issue was answered, closing this now.
[ ] I checked the documentation and related resources and couldn't find an answer to my question.
Your Question what is unclear to you? What would you like to know?
Code Examples This community speaks code. Share your code snippets to help us understand your question better.
Additional context Anything else you want to share with us?