search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.03k stars 53 forks source link

Evaluation with multiple GPUs #31

Open lucasjinreal opened 3 months ago

lucasjinreal commented 3 months ago

Currently using multiple GPU to eval, the model will load into serveral devices, caused inference error.

How to force them eval one task in parallel on multiple device?

kcz358 commented 3 months ago

Hi @lucasjinreal , you can refer to #12 for more details

lucasjinreal commented 3 months ago

@kcz358 Hi, the issue linked seems are runing one model with tensor parallel on 2 GPU.

But my request is doing evaluation same time on 8GPUs, my model is small and I want all GPUs runing at the same time, then gathering the results togather, does it support for now? How to enable that?

kcz358 commented 3 months ago

This seems like you should use accelerate launch --num_processes=8

lucasjinreal commented 3 months ago

Am using the main.py directly, since I need my customized llava, does there any other way didn't need to involve with accelerate?

kcz358 commented 3 months ago

If you does not want to use accelerate, you might also want to try torchrun. However, this is not tested and you might want figure out how to make it work.

lucasjinreal commented 3 months ago

Got it!

However, I met another problem, when eval mmbench, I got Nan result , while on cmmmu are normal:

num_fewshot: None,
batch_size: 1
|      Tasks       |Version|Filter|n-shot|  Metric  |Value|   |Stderr|
|------------------|-------|------|-----:|----------|-----|---|------|
|mmbench           |N/A    |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_cn_cc  |Yaml   |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_cn_dev |Yaml   |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_cn_test|Yaml   |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_en_dev |Yaml   |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_en_test|Yaml   |none  |     0|submission|N/A  |±  |N/A   |

|Groups |Version|Filter|n-shot|  Metric  |Value|   |Stderr|
|-------|-------|------|-----:|----------|-----|---|------|
|mmbench|N/A    |none  |     0|submission|N/A  |±  |N/A   |

any help?

kcz358 commented 3 months ago

Hi @lucasjinreal , you can see the metric is submission, which means you need to submit the files to their website by yourself to get the score

You can find the submission file in your log file

lucasjinreal commented 3 months ago

For mmbench, I saw the result file, it has prediciton and gt, why it can not calculate a score offline?

Luodian commented 2 months ago

For mmbench, I saw the result file, it has prediciton and gt, why it can not calculate a score offline?

MMBench is designed for submission only, you need to submit the result file to their official site.

lucasjinreal commented 2 months ago

No, for dev it's certainly not.

I am not refering to test.

Luodian commented 2 months ago

No, for dev it's certainly not.

I am not refering to test.

oh yes you're right. we will plan the update to enable report mmbench-dev score

lucasjinreal commented 2 months ago

thanks, hoping for it

hxhcreate commented 1 month ago

Run into the same problem when evaluation with multigpu, looking forward to any assistance, thanks

RROR Error Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution) in generating 05-10 21:54:48 [lmms_eval/models/llava.py:407] ERROR Error Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:3! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution) in generating