search

explodinggradients / ragas

Supercharge Your LLM Application Evaluations 🚀
https://docs.ragas.io
Apache License 2.0
7.06k stars 714 forks source link

Issues regarding ragas custom model #853

Closed shahules786 closed 4 months ago

shahules786 commented 6 months ago

Hi folks, please report issues regarding the ragas custom critic model in this thread.

Docs : https://docs.ragas.io/en/latest/howtos/customisations/ragas_custom_model.html Model: https://huggingface.co/explodinggradients/Ragas-critic-llm-Qwen1.5-GPTQ

valer1435 commented 6 months ago

Hi there! Thank you for your work!

I have a little question - is it possible to use this critic model for evaluation (not for test data generation)? I mean pass it as evaluate(llm=critic_llm), for example to calculate faithfulness.

If not, do you have plans to fine-tune model to replace gpt-4 in evaluation?

shahules786 commented 6 months ago

Hey @valer1435 the critic model is now fine-tuned only to act in test generation.

projects-g commented 6 months ago

Why is it so hard to find a full working example of how to use a custom LLM for test set generation ?

jjmachan commented 6 months ago

I'm assuming this didn't help @projects-g -> https://docs.ragas.io/en/stable/howtos/customisations/ragas_custom_model.html

where are you stuck at?