search

open-compass / MMBench

Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"
Apache License 2.0
163 stars 10 forks source link

Confusion about mmbench testset v1 vs v1.1 #37

Closed khiemkhanh98 closed 1 month ago

khiemkhanh98 commented 2 months ago

The data at Download Link (Legacy) at your front page is for test set v1 or v1.1? When i submit it on the server, it just shows me the result with test set without any explicit version(v1 or v1.1). Also, benchmarking it on VLMEvalKit is not possible for me since i am building my custom model and it is not supported yet in the toolkit, therefore i have to run the inference on this files myself.

kennymckormick commented 2 months ago

Hi, @khiemkhanh98 ,

  1. The version of the evaluation results depends on your submitted file version. If you submit v1.1, you will get results for v1.1. We will try to add a version information in the returned information.
  2. Actually, it's easy to integrate your ongoing VLM into the VLMEvalKit: you can maintain a private fork and incorporate your custom VLM. You just need to create a wrapper class and support the generate_inner() interface. More details in https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/advanced_guides/Development.md
khiemkhanh98 commented 2 months ago

Yes i got it. But the one at here: http://opencompass.openxlab.space/utils/MMBench/MMBench_TEST_EN_legacy.tsv is version 1 or 1.1 of the test set?

kennymckormick commented 2 months ago

legacy is for version 1