search

OpenLMLab / LEval

[ACL'24 Outstanding] Data and code for L-Eval, a comprehensive long context language models evaluation benchmark
GNU General Public License v3.0
343 stars 14 forks source link

Clarification regarding multidoc_qa #18

Open ogencoglu opened 1 month ago

ogencoglu commented 1 month ago

Can you elaborate on the format of the multidoc_qa data : https://huggingface.co/datasets/L4NLP/LEval/viewer/multidoc_qa

Is this simulating a chat setting where first question is "How long will Driver's Ed courses be valid for?" and then the other questions are follow up or are these independent questions?

[ "How long will Driver's Ed courses be valid for?", "I just want to know as I'm leaving the country for a few months and don't want to have to do it iagain.", "How would I be able to get this mailed to me?", "What if I need to change my name? Am I able to do that?", "What happens if I am visiting temporarily?" ]

ChenxinAn-fdu commented 1 month ago

Good question! You are right that this is a chat setting. Inputting all dialogue history would be the best choice, but it also poses challenges to the model.