search

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support 160+ VLMs, 50+ benchmarks
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Apache License 2.0
1.34k stars 188 forks source link

About Tablevqabench score #442

Closed mary-0830 closed 1 month ago

mary-0830 commented 2 months ago

Hello, may I ask which scores the TableVQAbench output should correspond to in the paper? image

image And where should Input Modalities be set?

Thanks for your reply~

FangXinyu-0913 commented 2 months ago

Hi @mary-0830, For TableVQABench, as of now VLMEvalKit only supports evaluating MLLMs, and the output modal only supports Vision, the specific four values that correspond to the four values of the same name on the table. The calculation of the average value is a bit off from my previous knowledge, maybe ask @hkunzhe.