There is an inconsistency in the number of mmbench images

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval

https://lmms-lab.github.io/

Other

1.02k stars 52 forks source link

There is an inconsistency in the number of mmbench images #116

Open MuskAI opened 1 week ago

MuskAI commented 1 week ago

According to the images downloaded by the LLaVA 1.5 evaluation tool, the number of images in mmbench en dev is 4377, while that of lmms-lab/MMBench is only 4329, which may be the cause of the discrepancy between the two evaluation results.

I want to know why there are some pictures missing in huggingfacelmms-lab/MMBench

Thank you for your contributions!

Luodian commented 1 week ago

Thanks! I am aware of that MMBench has been updated by the authors during the rebbutal and resubmission from CVPR to ECCV. So I am not sure if we used version is newer than LLaVA 1.5's

I think the MMBench evaluation set from old to new is:

LLaVA 1.5 used version
LMMs-Eval used version
MMBench-v0.2

I will take deep look into this issue.