Issue with TestsetGenerator.generate_with_langchain_docs: Frequent errors and clarification requested

[donbr]

First, I'd like to express my appreciation for this excellent cookbook repository. It's an invaluable resource for demonstrating the effective integration of Qdrant with Ragas and language models, and it's incredibly helpful for customers looking to implement effective RAG evaluations.

I'm encountering an issue while working with the TestsetGenerator, specifically when using the generate_with_langchain_docs method in the workshop-rag-eval-qdrant-ragas/notebook/naive_rag_eval_qdrant_ragas.ipynb notebook. My notes are captured in the Test Evaluation Dataset Generation using Ragas section.

Here are the details:

1. Problem Description: The generate_with_langchain_docs method is failing with errors approximately 80% of the time.

2. Code Snippet:

   generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)
   testset = generator.generate_with_langchain_docs(
      langchain_docs[:10], 
      test_size=10, 
      distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}
   )

3. Questions: a. Could this be related to a specific Langchain version? If so, which version is recommended for use with this cookbook?

   b. I noticed in the Ragas documentation (https://docs.ragas.io/en/stable/howtos/customisations/azure-openai.html#test-set-generation) that it's recommended to set raise_exceptions and with_debugging_logs to False. Should these parameters be applied in this context as well?

   c. If we do set these parameters to False, what would be the impact on error handling and debugging? Are there any potential drawbacks to consider?

4. Environment Details:

   • Ragas version: [Please specify]
   • Langchain version: [Please specify]
   • Python version: [Please specify]

Any insights or guidance on resolving this issue would be greatly appreciated. Thank you for your time and for maintaining this valuable resource.

[jjmachan]

hey @donbr which models are you using and which version of Ragas. The latest version (0.1.10) has some fixes for these issues, could you test it out?

[atarora]

Hi @donbr ,

Thank you for checking in and detailed description. I wanted to ensure that you checked : https://github.com/qdrant/qdrant-rag-eval/blob/master/workshop-rag-eval-qdrant-ragas/pyproject.toml Each of these notebooks and folders are managed by Poetry for version management and details here (https://github.com/qdrant/qdrant-rag-eval/tree/master/workshop-rag-eval-qdrant-ragas) on how to install / configure your environment.

I retried (and I checked with RAGAS team too, to be extra sure) that the error max retries exceeded for SimpleEvolution (if that was the concern) is intended but this does not affect the question generation , if you execute next block , you will see that the questions were generated as expected.

I hope that helps , if not, don't hesitate to write back ! Thank you !