search

open-compass / MMBench

Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"
Apache License 2.0
164 stars 10 forks source link

The Circular Evaluation Strategy && LLM-based Choice Extractors #8

Closed yyy809643573 closed 11 months ago

yyy809643573 commented 1 year ago

hi, i am using the MMBench to evaluate my VLM, but i find there are some ploblems in MMBench

  1. I see that in https://github.com/open-compass/opencompass/blob/main/configs/multimodal/minigpt_4/README.md, the minigpt4 does not achieve the The Circular Evaluation Strategy and LLM-based Choice Extractors by using chatgpt，Are The Circular Evaluation Strategy and LLM-based Choice Extractors needed to achieve by ourselves？
  2. the question number of MMBench-Dev(En) is not 1164 in describtion of https://github.com/open-compass/MMBench

Looking forward to your reply，Thank you！

kennymckormick commented 1 year ago
  1. CircularEval and LLM-based Choice Extractors are implemented in eval_mmbench.py, which is not included in the inference part.
  2. The question number in the MMBench-Dev tsv is more than 1164, since it also includes CircularEval cases with choice shifted.