(spike) Identify Open Source Datasets for RAG Evaluations

[jalling97]

Description

We need to choose a small number (1-3, depending on size) of open source RAG evaluation datasets. Having at least 1 open source dataset allows us to begin running basic evaluations that allow us to begin iterating on improvements.

Relevant Links

Many datasets can be found on HuggingFace

[jalling97]

A few options/references to take a look at:

• https://huggingface.co/datasets/explodinggradients/amnesty_qa
• https://github.com/Arize-ai/phoenix/blob/main/tutorials/evals/evaluate_rag.ipynb
• https://github.com/Arize-ai/phoenix/blob/main/tutorials/ragas_retrieval_evals_tutorial.ipynb

[jalling97]

Relevant to here: https://github.com/defenseunicorns/leapfrogai/issues/716#issuecomment-2214386582

[jalling97]

Based on popular guides for setting up RAG datasets, most lean towards performing synthetic dataset generation, rather than sharing out specific datasets that contain questions, answers, and context. Going forward, it will likely make the most sense to make some sample datasets using the contextual documents and generating them. Some potential options include:

• https://www.whitehouse.gov/wp-content/uploads/2024/04/Global-Health-Security-Strategy-2024-1.pdf
• https://www.armed-services.senate.gov/imo/media/doc/fy24_ndaa_conference_executive_summary1.pdf
• https://www.humanesociety.org/sites/default/files/docs/HSUS_ACFS-2023.pdf
• https://dodcio.defense.gov/Portals/0/Documents/Library/(U)%202024-01-02%20DoD%20Cybersecurity%20Reciprocity%20Playbook.pdf

[jalling97]

For Needle In a Haystack (NIAH) evaluations, we can leverage some existing datasets for this, to potentially include:

• https://huggingface.co/datasets/nanotron/simple_needle_in_a_hay_stack
• https://huggingface.co/datasets/lvwerra/needle-llama3-16x512 (similar setup to the nanotron dataset, but split into multiple datasets based on context length)

[jalling97]

These proved to be pretty good, but I think it would be more advantageous to generate some simple examples instead.

[jalling97]

Final determination:

Of the open source datasets available, it was found to be more valuable to create a few datasets from scratch, so no open source datasets will be used for RAG evaluations specifically.