Error during videomme - Githubissues

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval

https://lmms-lab.github.io/

Other

1.02k stars 52 forks source link

Error during videomme #112

Open yukang2017 opened 1 week ago

yukang2017 commented 1 week ago

Hi,

When testing models on llava model and videomme benchmarks, there is one error when loading videos. The process_images function can not read mp4 files.

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/2797987f5b88b87bd172714b678a75a1d8051826/lmms_eval/models/llava.py#L326

Luodian commented 1 week ago

@choiszt Can you help to take into look at this issue? Thanks!

choiszt commented 1 week ago

Hi @yukang2017,

Thank you for your interest in the LMMS-Eval and Videomme benchmarks!

Regarding your question, the LLaVA model is specifically designed for image benchmark evaluations. For supported video models, I would recommend referring to the following resources:

I hope you find these resources helpful for your work on video benchmarks.