How to create RAG Evaluation Test Data to use in InspectorRAGet

[swaroop11]

Hi Team, Thanks for designing RAG Evaluation package this was a much needed thing in LLMs projects.

I wanted to know how can I create/convert my dataset into the format which I can use in InspectorRAGet? And what are the precesses/steps I need to perform before putting in for visualisations?

I have a excel sheet with Questions, GT Answers, GT RAG Contexts as my test dataset, what the other parameters required?

Thanks in Advance

[kpfadnis]

Hello @swaroop11 Thanks for the interest.

InspectorRAGet is primarily useful to see performance of different models on your RAG tasks. So it excepts following details at minimum

1. Models being evaluated
2. Metrics used for evaluations. Example BLEU, Rouge-L, Faithfulness score etc.
3. Tasks
4. Evaluations

Please see a reference JSON (https://github.com/IBM/InspectorRAGet/blob/main/data/clapNQ.json) to get sense of how to convert your excel into InspectorRAGet input format.

We also have a few notebooks that can help you run a sample experiments with HuggingFace or LLMEvalHarness here (https://github.com/IBM/InspectorRAGet/tree/main/notebooks).