search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.02k stars 52 forks source link

Adding microsoft/Phi-3-vision-128k-instruct model. #87

Closed vfragoso closed 1 month ago

vfragoso commented 1 month ago

Description:

  1. Adding the newly announced Phi-3 Vision model into the benchmark.
  2. Model reproduces reported numbers:
Evaluation MMMU ScienceQA-Img MathVista ChartQA AI2D TextVQA POPE MMB
LMMs-Eval 40.7 90.6 44.4 81.4 77.1 69.4 86.2 gpt=73.7 / plain_accuracy=80.5
Reported 40.4 90.8 44.5 81.4 76.7 70.9 85.8 plain_accuracy=80.5

For the results shown above, we used 4 RTX A600 GPUs.