When testing LLaVA-v1.5 with eval.py, the following error occurs.
*** RuntimeError: Expected all tensors to be on the same device, but found at least two devices,
cuda:0 and cuda:1! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
This is because when using huggingface to load the model, the default parameter device_map="auto", the model will be loaded to multiple GPUs (Pipeline Parallelism).
With the AlignDevicesHook conflict, the data is loaded to other gpus in some layer, and now all the parameters are on the default gpu, which triggers the error report.
Problems
When testing LLaVA-v1.5 with eval.py, the following error occurs.
This is because when using huggingface to load the model, the default parameter device_map="auto", the model will be loaded to multiple GPUs (Pipeline Parallelism).
While in eval.py, the wrapped model(
MLLM_Tester
) will be called thecuda
method, and the model parameters will be loaded to the default gpu again.https://github.com/AILab-CVC/SEED-Bench/blob/fbc5f2c98eb4ce8be40a9e5a9e55f767d7a45a0b/eval.py#L171
With the AlignDevicesHook conflict, the data is loaded to other gpus in some layer, and now all the parameters are on the default gpu, which triggers the error report.
Solution
I think removing
.cuda()
here is ok, though I only check the llava interface. https://github.com/AILab-CVC/SEED-Bench/blob/fbc5f2c98eb4ce8be40a9e5a9e55f767d7a45a0b/eval.py#L171