search

open-compass / MMBench

Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"
Apache License 2.0
164 stars 10 forks source link

Problem about Numbers of MMBench #12

Closed 1649759610 closed 10 months ago

1649759610 commented 1 year ago

image

This table lists the number of each split.

I download each split, such as MMBench Test(cn), and loaded with the script here, then count the examples. it is 6666, not 1784 listed in the above table.

Can you explain this for me , thanks.

mary-0830 commented 11 months ago

There is also this problem. Do you know what's going on?

kennymckormick commented 10 months ago

Hi, @1649759610 , #Questions indicates the real number of distinct questions in the datasets, while the number of lines in the tsv file also include the CircularEval passes (for example, 4 copy of a single question if it has 4 choices), so the line number is ~4x of the question number.