search

explodinggradients / ragas

Supercharge Your LLM Application Evaluations 🚀
https://docs.ragas.io
Apache License 2.0
7.14k stars 727 forks source link

Persist test set sample questions to TestSetGenerator object so as to not lose work #1479

Open ahgraber opened 3 weeks ago

ahgraber commented 3 weeks ago

Describe the Feature I am currently on a tier 1 account with Anthropic, which means I have a daily cap of 1M tokens (total - in and out). When trying to generate a testset of 100 questions, I hit that daily token limit and the generator.generate call raises a RateLimitError. However, the data is not preserved, which means I have now paid for 1M tokens but critically do not have access to the results.

Therefore, as a developer I would like the generated test questions to be persisted to the TestSetGenerator object as they are received so that I can access the testset up to the point that the Exception occurred. Even better would be to catch the Exception and enter a "pause" state so that the next day I could run something like generator.continue() and pick up where I left off.

Why is the feature important for you? See above

Additional context Add any other context about the feature you want to share with us.

shahules786 commented 3 weeks ago

Hey @ahgraber thanks for bringing this up. As part of 0.2 we are introducing new test generation algorithm. We will add this to our immediate roadmap for improvements after 0.2 release. https://docs.ragas.io/en/latest/getstarted/rag_testset_generation/

ahgraber commented 3 weeks ago

Amazing! I'm really looking forward to experimenting with v0.2

shahules786 commented 2 weeks ago

Hey @ahgraber It's out. Please let us know if you need any help with it.