Custom prompt/seed for question generation

[ableiweiss]

Is it possible to feed in a custom prompt into TestsetGenerator via the API? Alternatively, is it possible to modify the seed question prompt?

The only way I see of doing so is by modifying internal RAGAS code, would be useful to do so via the external API.

[dosubot[bot]]

Hi @ableiweiss! I'm a bot here to help you with any questions or issues you might have. Let's tackle this together!

Yes, it is possible to feed a custom prompt into TestsetGenerator via the API. You can modify the seed_question_prompt by defining a new Prompt object with your specific instructions and examples. Here is an example:

from ragas.llms.prompt import Prompt
from ragas.testset.generator import TestsetGenerator
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Define the custom prompt
custom_prompt = Prompt(
    name="custom_seed_question",
    instruction="Generate a custom question that can be fully answered from the given context. The question should be formed using the provided topic.",
    examples=[
        {
            "context": "Quantum computing leverages the principles of quantum mechanics to perform computations at speeds unattainable by classical computers.",
            "keyphrase": "Quantum computing",
            "question": "What are the advantages of quantum computing over classical computing?",
        },
        {
            "context": "Machine learning algorithms enable computers to learn from data and make decisions based on patterns and insights.",
            "keyphrase": "Machine learning",
            "question": "How do machine learning algorithms improve decision-making processes?",
        },
    ],
    input_keys=["context", "keyphrase"],
    output_key="question",
    output_type="str",
)

# Initialize the TestsetGenerator with the custom prompt
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

# Replace the default prompt with the custom prompt
simple.seed_question_prompt = custom_prompt

# Generate test sets using the custom prompt
documents = load_your_documents()  # Replace with your document loading logic
testset = generator.generate_with_langchain_docs(documents, 10, {simple: 1.0}) 
testset.to_pandas()

This example demonstrates how to create a custom prompt and integrate it into the TestsetGenerator class for generating test sets ^[1].

To continue talking to Dosu, mention @dosu.

[github-actions[bot]]

It seems the issue was answered, closing this now.