search

explodinggradients / ragas

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines
https://docs.ragas.io
Apache License 2.0
6.34k stars 609 forks source link

[R-241] docs on how to custumize LLMs in test set generation #856

Closed projects-g closed 3 weeks ago

projects-g commented 4 months ago

[x] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question Was anyone able to generate synthetic dataset using a custom LLM ? The documentation completely skips over the specifics here. It just asks us to either create a langchain instance (supported models are only few) and wrap it around new wrappers OR base them out of BaseRagasLLM https://docs.ragas.io/en/stable/howtos/customisations/bring-your-own-llm-or-embs.html#llms

But when you do that, you also have to define agenerate_prompt, agenerate_text, generate_prompt and generate_text. We dont know when these functions are called by the test set generation and what to return at each function, etc. Its all a black box with no examples. My experiments tend to repeat multiple times with no output and just fails after a while.

Code Examples This community speaks code. Share your code snippets to help us understand your question better.

Additional context Anything else you want to share with us?

R-241

WGS-note commented 3 months ago

hi，have you solved it?

projects-g commented 3 months ago

Unfortunately, I did not receive a solution to this yet.

Zzrjin commented 1 month ago

Any news about the solutions now? Facing the same problem.

jjmachan commented 3 weeks ago

@projects-g @WGS-note @Zzrjin extremely sorry about the delay but we had improved the docs for this

take a look at https://docs.ragas.io/en/stable/howtos/customisations/bring-your-own-llm-or-embs.html#test-set-generation

and it should help

closing this for now but feel free to comment below if you have any issues and I'll reopen