Review the licenses for all datasets

[davidsbatista]

• [X] The ARAGOG dataset.
• [X] The SQuAD dataset.
• [ ] The Mini ESG Bench data are public documents - need to find the source
• [ ] Braintrust, no idea - I think we can also remove it for now

[davidsbatista]

Regarding the 2 datasets for which we already have evaluation code working.

The ARAGOG dataset:

• is a subset from this set: https://huggingface.co/datasets/jamescalam/ai-arxiv
• See Section 3.1 (https://arxiv.org/pdf/2404.01037): "This study utilizes a tailored dataset derived from the AI ArXiv collection, accessible via Hugging Face (James Calam, 2023). The dataset consists of 423 selected research papers centered around the themes of AI and LLMs, sourced from arXiv. This selection offers a comprehensive foundation for constructing a database to test the RAG techniques and creating a set of evaluation data to assess their effectiveness."
• we are using the questions/answers pairs which were human-validated, we should cite the paper, and that should be enough.

The SQuAD dataset:

• is distributed under the CC BY-SA 4.0 license
• we need to reference the original source: web page, paper

[mrm1001]

For the mini ESG dataset:

• source files are here: https://github.com/run-llama/llama-datasets/tree/main/llama_datasets/mini_esg_bench/source_files

The main files are all public:

• https://news.microsoft.com/wp-content/uploads/prod/sites/42/2023/05/2022-Environmental-Sustainability-Report.pdf

• https://sustainability.aboutamazon.com/2022-sustainability-report.pdf

• https://www.apple.com/environment/pdf/Apple_Environmental_Progress_Report_2022.pdf

• https://sustainability.fb.com/wp-content/uploads/2022/06/Meta-2021-Sustainability-Report.pdf

• https://downloads.ctfassets.net/4cd45et68cgf/7rnC6zK537cM8zAGrXA90E/3c654a2d0023a4dac26a20b2fff39855/Netflix_2022-ESG-Report-FINAL.pdf

• https://sustainability.google/reports/google-2022-environmental-report/

• and the question-answer pairs are here: https://github.com/run-llama/llama-datasets/blob/main/llama_datasets/mini_esg_bench/rag_dataset.json

Issue opened on the llama-datasets repo: https://github.com/run-llama/llama-datasets/issues/54