The answer set here is a 100% wrong.

Psycoy / MixEval

The official evaluation suite and dynamic data release for MixEval.

https://mixeval.github.io/

222 stars 34 forks source link

The answer set here is a 100% wrong. #8

Closed XapaJIaMnu-at-meta closed 4 months ago

XapaJIaMnu-at-meta commented 4 months ago

https://github.com/Psycoy/MixEval/blob/03ee6e606d3b5af8fdb2b1da711f5672d0c98482/mix_eval/data/mixeval-2024-06-01/mixeval/free-form.json#L672

The answerset is a bunch of buzzwords from the harry potter books, it should be just Hufflepuff

Psycoy commented 4 months ago

Hi Nikolay, thanks for proposing this issue.

This is the problem of the original TriviaQA annotation. Such annotation errors have a negligible number, so they won't affect the result significantly.