search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.42k stars 119 forks source link

The evaluation of videoMME is very slow #134

Closed xiaofei05 closed 2 months ago

xiaofei05 commented 3 months ago
Luodian commented 3 months ago

@choiszt @ZhangYuanhan-AI Do you have any idea about it? 😜

ZhangYuanhan-AI commented 3 months ago

@xiaofei05

Please kindly give us more information:

  1. the command you used
  2. The model you used

From our side, the evaluation time for a 7B model on video-mme should be around 15 minutes with 8 GPU.

liguopeng0923 commented 2 months ago

I met the same problem. I found that when I set the num_process as 1, the pre-process is very slow.

image
liguopeng0923 commented 2 months ago

When I use num_process with 2, the speed is normal. But the videomme is not registered in All_tasks.

image image
liguopeng0923 commented 2 months ago

When I use num_process with 2, the speed is normal. But the videomme is not registered in All_tasks.

image image

I solve it by export HF_HOME="~/.cache/huggingface/"

xiaofei05 commented 2 months ago

I've re-cloned the latest version of the code, and the issue is now resolved.

exiawsh commented 1 month ago

@liguopeng0923 Hi. How to solve the promble? I alse came cross the videomme is not registered in All_tasks.