search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.36k stars 103 forks source link

video-mme (short/medium/long) #254

Open orrzohar opened 5 days ago

orrzohar commented 5 days ago

Hi,

Currently, video-mme only outputs an overall score, without the short/medium/long video breakdown. How can I get this?

Best, Orr

Luodian commented 5 days ago

@choiszt Hi Shuai, can you help to check this?

choiszt commented 4 days ago

Hi Orr,

Regarding the breakdown of short, medium, and long video types (VIDEO_TYPE), as well as the CATEGORIES and SUB_CATEGORIES, we currently output these directly by printing them to the terminal.

If you’re facing issues with terminal output due to multi-node setups, you might consider redirecting the terminal output to a .txt file, which would allow you to check specific fields more easily.

For more details on how we handle the evaluation output, you can refer to our implementation here.

Best, Shuai

orrzohar commented 4 days ago

Hi @choiszt,

I am doing a lot of evals (1000s), so am using the results.json as opposed to the terminal printouts to automatically collect the evaluation results.

Is there an easy way I can change the write out to have all these in the results.json? could you point me to where I would need to make the changes?

Best, Orr

Luodian commented 4 days ago

Hi Orr, here's code related to saving results.json

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/e1f9adefcc4b92a39edb2450fbadf659a84364fb/lmms_eval/__main__.py#L511-L537

Luodian commented 4 days ago

I was considering making a PR to fix the output_path issue, since it's mentioned many times.