How are the image types defined and labeled?

MMMU-Benchmark / MMMU

This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI"

https://mmmu-benchmark.github.io/

Apache License 2.0

327 stars 21 forks source link

How are the image types defined and labeled? #6

Closed CCYChongyanChen closed 9 months ago

CCYChongyanChen commented 9 months ago

Great work! Could I ask how are the image types defined and labeled? Thanks!

xiangyue9607 commented 9 months ago

Hi,

Thanks for the question. We first compiled a long list covering different types of images in different subjects with the help of GPT-4 and other information online. We then showed the list to the annotators and let them select one or more image types for the image(s) in each example. If none of them matched the pre-defined list, they would mark it as "Other".

Hope it helps!

Best, Xiang

xiangyue9607 commented 9 months ago

close the issue. let us know if you have further questions