search

OpenLMLab / LEval

[ACL'24 Oral] Data and code for L-Eval, a comprehensive long context language models evaluation benchmark
GNU General Public License v3.0
314 stars 13 forks source link

There are some minor mistakes in the dataset #1

Closed wtangdev closed 1 year ago

wtangdev commented 1 year ago

Hi! Firstly, I want to express my appreciation for the great work on the dataset designed for the long text challenge; it's a valuable resource, and I'm really glad to see it.

However, there are some minor mistakes that exist. For example, in the first sample of load_dataset('L4NLP/LEval', "coursera", split='test'), the Question 4 occurs twice. According to my test, although this is a minor mistake, it may influence the performance of the experimental results. So I want to point it out~

Looking forward to your update. Have a good day~

ChenxinAn-fdu commented 1 year ago

Hi!! Thank you so much. It seems that there is something wrong with the question index, but fortunately, only the index is incorrect, the questions themselves were not. Since we only feed one query at a time, it shouldn't affect the results too much. We have already corrected this issue. Once again, thank you~