explodinggradients / ragas

Supercharge Your LLM Application Evaluations 🚀
https://docs.ragas.io
Apache License 2.0
7.33k stars 747 forks source link

Can the generated dataset be filled in the blank or true/false questions #1620

Open SummerDream233 opened 3 weeks ago

SummerDream233 commented 3 weeks ago

I notice that the generated test dataset is Q&A, and it means that there is a question and an answer.

I want to know whether I can use Ragas to generate other typies of queston, such as fill in the blank, true/false, or multiple-choice questions.

If it can do it, how should I write the code? Because I didn't see any relevant settings in the document.

amabilee commented 3 weeks ago

Unfortunately I didn't find concrete answers on the internet but here is some information that might help you.

Here it says

From true/false and multiple choice questions all the way to the opinion-based, interpretation, comparison and critical thinking questions.

And in this issue where the problem actually has a question_type atribute

"quiz": [
{
"question": "True or False",
"answer": "false",
"question_type": "true_or_false"
}
shahules786 commented 2 weeks ago

Hey @SummerDream233 Yes, this can be done. Let me add to the documentation issues and create a doc on how to do it this week itself.