note inclusion of prev Q&A pairs in prompt for SQuAD eval in eval_det… - Githubissues

meta-llama / llama-models

Utilities intended for use with Llama models.

Other

4.92k stars 843 forks source link

note inclusion of prev Q&A pairs in prompt for SQuAD eval in eval_det… #85

Closed jasonkrone closed 3 months ago

jasonkrone commented 3 months ago

Change:

add a note in the llama3_1 eval_details.md file mentioning that for SQuAD prev Q & A pairs pertaining to a given passage are included in the prompt.

Motivation:

To the best of my knowledge this evaluation detail isn't mentioned publicly except for in the eval details dataset. My hope is that also including it in the eval_details.md will help other researchers catch this important detail without having to dive deep into the eval details dataset.