search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
2.07k stars 153 forks source link

llava_hf inference is extremely slow #387

Open luomancs opened 3 weeks ago

luomancs commented 3 weeks ago

Hi there,

Thank you for the benchmark. I have tried to this repo and get the inference of llava_hf model type and the checkpoint of llava-hf/llava-v1.6-mistral-7b-hf on infovqa, however, the inference speed is extremely slow and also it get out of memory issue even when I use A100 gpu with 80G memory. The following is my command line:

python -m accelerate.commands.launch \ --num_processes=8 \ -m lmms_eval \ --model llava_hf \ --model_args pretrained="llava-hf/llava-v1.6-mistral-7b-hf" \ --tasks infovqa \ --batch_size 1 \ --log_samples \ --log_samples_suffix llava_v1.6_mistral_infovqa_scienceqa_docvqa \ --output_path ./logs/

On the other hand, running the inference with llava model type with checkpoint of liuhaotian/llava-v1.6-mistral-7b on the same dataset only requires 45 mins to do the inference without any OOM issue.

Thanks for your guidance.

kcz358 commented 3 weeks ago

There is a fix recently in #386 to fix some issues for llava_hf that possibily relate to this issue. Can you try that branch?