search

OpenGVLab / MM-NIAH

[NeurIPS 2024] Needle In A Multimodal Haystack (MM-NIAH): A comprehensive benchmark designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.
https://mm-niah.github.io/
79 stars 4 forks source link

Confusion about results on reasoning-image task #1

Open yayafengzi opened 3 months ago

yayafengzi commented 3 months ago

The reasoning-image task only have two choices, so "random choose" can get 50 scores. However, performance of the best model "InternVL-Chat-V1-5-RAG" is image Does it means this task is too difficult?

Weiyun1025 commented 3 months ago

Thank you for your interest in our benchmark.

The main reason for the underperformance of these models, as compared to random guessing, is their inability to understand images presented as options. For example, when we provide a question in the format <question>\nA.<image>\nB.<image>, the models often do not respond with either A or B, but instead generate a lengthy, irrelevant passage.

We argue that this task is not hard, as humans can easily achieve near-perfect performance. On the contrary, we think this task effectively demonstrates the current limitations of multimodal models in understanding multiple images.