TestsetGenerator fails with "AttributeError: 'str' object has no attribute 'generator_llm'"

[vecorro]

[X] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug Hi,

I tried to generate a test dataset with TestsetGenerator from both LangChain and LlamaIndex documents and I'm hitting the same issue. It looks like it is due to a bad attribute transfer across classes where the generator LLM object gets replaced by a str. Please f ind the details below.

Thanks very much!

Ragas version: 0.1.4 Python version: 3.10

Code to Reproduce

from llama_index.core import SimpleDirectoryReader
from llama_index.readers.file import PyMuPDFReader
from langchain_community.llms import VLLMOpenAI
from langchain_community.embeddings import HuggingFaceEmbeddings
from ragas.testset import TestsetGenerator

PDF_FILES_PATH = "../../doc_collections"

EMBEDDING_MODEL = "BAAI/bge-base-en-v1.5"
model_kwargs = {'device': 'cuda:0'}
encode_kwargs = {'normalize_embeddings': False}
langchain_emb = HuggingFaceEmbeddings(
    model_name=EMBEDDING_MODEL,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)

langchain_llm = VLLMOpenAI(
    openai_api_base="localhost/api/v1",
    model_name="WizardLM/WizardLM-70B-V1.0",
    temperature=0.1,
    max_tokens=512,
)

%%time
# Lamda function to add the file name as metadata at loading time
filename_fn = lambda filename: {"file_name": filename.split("/")[-1]}

reader = SimpleDirectoryReader(
    input_dir=PDF_FILES_PATH,
    required_exts=[".pdf"],
    file_extractor={".pdf":PyMuPDFReader()},
    file_metadata=filename_fn,
    num_files_limit=10,
)
documents = reader.load_data()

dataset_distribution = {
    "simple": 0.4,
    "reasoning": 0.25,
    "multi_context": 0.1,
    "conditional": 0.25,
}

dataset_generator = TestsetGenerator.from_langchain(
    generator_llm=langchain_llm,
    critic_llm=langchain_llm,
    embeddings=langchain_emb,
    #docstore=doc_store,
)

dataset = dataset_generator.generate_with_llamaindex_docs(
    documents=documents[:25],
    test_size=10,
    distributions=dataset_distribution,
)   

Error trace

Filename and doc_id are the same for all nodes.

--------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 15
      1 dataset_distribution = {
      2     "simple": 0.4,
      3     "reasoning": 0.25,
      4     "multi_context": 0.1,
      5     "conditional": 0.25,
      6 }
      8 dataset_generator = TestsetGenerator.from_langchain(
      9     generator_llm=langchain_llm,
     10     critic_llm=langchain_llm,
     11     embeddings=langchain_emb,
     12     #docstore=doc_store,
     13 )
---> 15 dataset = dataset_generator.generate_with_llamaindex_docs(
     16     documents=documents[:25],
     17     test_size=10,
     18     distributions=dataset_distribution,
     19 )   

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/ragas/testset/generator.py:152, in TestsetGenerator.generate_with_llamaindex_docs(self, documents, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    147 # chunk documents and add to docstore
    148 self.docstore.add_documents(
    149     [Document.from_llamaindex_document(doc) for doc in documents]
    150 )
--> 152 return self.generate(
    153     test_size=test_size,
    154     distributions=distributions,
    155     with_debugging_logs=with_debugging_logs,
    156     is_async=is_async,
    157     run_config=run_config,
    158     raise_exceptions=raise_exceptions,
    159 )

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/ragas/testset/generator.py:227, in TestsetGenerator.generate(self, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    225 # init filters and evolutions
    226 for evolution in distributions:
--> 227     self.init_evolution(evolution)
    228     evolution.init(is_async=is_async, run_config=run_config)
    230 if with_debugging_logs:

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/ragas/testset/generator.py:189, in TestsetGenerator.init_evolution(self, evolution)
    188 def init_evolution(self, evolution: Evolution) -> None:
--> 189     if evolution.generator_llm is None:
    190         evolution.generator_llm = self.generator_llm
    191         if evolution.docstore is None:

AttributeError: 'str' object has no attribute 'generator_llm'

Expected behavior I expected a successful generation of the dataset

Additional context The same error occurs when I try to generate from langchain or llamaindex docs.

[shahules786]

Hey @vecorro , there is a mistake in your code

from ragas.testset.evolutions import simple, reasoning, multi_context
distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}

evolutions are objects and not strings. Which documentation did you follow? It could be an older one which we forgot to update.

[shahules786]

Also a recommendation, it would be better to use mixtral 7*8 than models from wizard ml (derivatives of llama-2) @vecorro

[vecorro]

Many thanks @shahules786 you are right. I took the example from another user commenting on other issues. I re-checked the existing online document on Generate a Synthetic Test Set and looks like the code block you shared with me. The error went away although the generation process has over 2hr running. I requested a 'test_size=10' over a 100 documents corpus. I still don't get the exact meaning of the 'test_size' parameter but the documentation does not provide an explanation about it. Anyways I'm closing this issue as the error I reported was due to a misconfiguration from my side.

[vecorro]

BTW, you were right. The Wizard LLM was causing errors in the generation process. I switched to Zephyr which is derived from Mistral 7b and seems to be working, or at least not generation execution errors :) Many, many thanks