Corrupted MathVista testmini data

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support 160+ VLMs, 50+ benchmarks

https://huggingface.co/spaces/opencompass/open_vlm_leaderboard

Apache License 2.0

1.34k stars 188 forks source link

Corrupted MathVista testmini data #578

Open LeoDu0314 opened 1 week ago

LeoDu0314 commented 1 week ago

I found that the MathVista testmini data used in VLMEvalkit(https://opencompass.openxlab.space/utils/VLMEval/MathVista_MINI.tsv) is not identical to that provided by the MathVista offical huggingface repo(https://huggingface.co/datasets/AI4Math/MathVista)

The VLMEvalkit version of MathVista testmini data has the following problem:

Images are compressed
Misuse of \r\n in some questions.

Could you provide an updated version of MathVista testmini?

kennymckormick commented 1 week ago

Hi, @LeoDu0314 , During our pre-processing, we have converted images to JPEG format. We will try to build another version that do not perform the JPEG conversion and do the evaluation again to see the gap.