search

OpenBMB / OlympiadBench

[ACL 2024]Official GitHub repo for OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems.
MIT License
92 stars 7 forks source link

Questions about the dataset and code #4

Closed zxy-smart closed 1 month ago

zxy-smart commented 7 months ago

Could you clarify the main differences between the new dataset and the previous one? Are there any significant changes in the data format, or anything else that I should be aware of?

During my review of the project contents, it appears that files named 'YI_VL_Evaluator.py' and 'YI_Chat_Evaluator.py' are missing. Could you provide an example or let me know where I can find it? Thank you for your assistance and support.

Hothan01 commented 7 months ago

There are two main changes to the dataset:

  1. We make adjustments to the data format, mainly key value changes, which have no impact on usage. Also, for the images, we provide local versions instead of links.
  2. We rewrite some of the previous open-ended problems that could not be automatically assessed by experts. The open-ended problems are now available for automatic evaluation. And the documents are divided according to the category characteristics. The above changes are taken into account in the inference code.

We will update these two code files later, please stay tuned.