search

AILab-CVC / SEED-Bench

(CVPR2024)A benchmark for evaluating Multimodal LLMs using multiple-choice questions.
Other
275 stars 8 forks source link

VLMs vs LLMs evaluation #12

Open idan-tankel opened 8 months ago

idan-tankel commented 8 months ago

Hello 👋

First of all thank you for the great work and evaluation results!

I have understood that in many cases you predicted outputs for each question based on the choice that minimizes the loss of the current evaluated model. I wanted to ask - if there was any difference between the evaluation for VLMs and LLMs? and if was, how did you put these results on the same scale?

many thanks Idan

Bohao-Lee commented 6 months ago

Thank you for your interest in our work, and we apologize for the delayed response. In reality, there is no difference in the evaluation of VLMs and LLMs. The only distinction is that LLMs take only the question as input, while VLMs consider both the question and its corresponding image. We simply calculate the loss for each candidate.