search

JUNJIE99 / MLVU

🔥🔥MLVU: Multi-task Long Video Understanding Benchmark
153 stars 0 forks source link

How to understand the 'Input' in Leaderboard ? #8

Open jchsun1 opened 2 days ago

jchsun1 commented 2 days ago

Hello，In the evaluation Leaderboard, the value of 'Input' column is usually 'n frm', '16 frm' for example. How do I understand this value? Is 16 frames sampled from the entire video as input for the end-to-end inference? If so, 16 frames contain too little content compared to a video? Looking forward to your reply.

JUNJIE99 commented 2 days ago

"16frm" refers to the 16 frames extracted from a video and fed into a Multi-Modal Language Model (MLLM). Our evaluation method follows the original design of each model, with most models using uniform sampling.

While 16 frames are insufficient for understanding long videos, this was due to the early stage of Video MLLM technology at that time. Recently, however, many new Video MLLMs specifically designed for long videos have emerged. These new models can process more input frames and perform better, as evidenced by their high rankings on leaderboards.