Wrongly treating mvbench as an image task

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval

https://lmms-lab.github.io/

Other

1.44k stars 124 forks source link

Wrongly treating mvbench as an image task #205

Open hulianyuyy opened 1 month ago

hulianyuyy commented 1 month ago

When i evaluate llava onevision on the video benchmark "mvbench", the task regards it as an image task. Please fix it.

exiawsh commented 1 month ago

It seems that there is an Image in mvbench? container = av.open(video_path) <PIL.Image.Image image mode=RGB size=640x360 at 0x15531C163F70>
ValueError: Could not determine output format

YangYangGirl commented 1 month ago

I met the some question.

YangYangGirl commented 1 month ago

Probably because this video is missing: ~/.cache/huggingface/mvbench/star/Charades_v1_480/EDXBD.mp4