search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.02k stars 52 forks source link

Support for llava-hf/llava-v1.6-x #74

Open justinphan3110cais opened 1 month ago

justinphan3110cais commented 1 month ago

Hi,

I'm trying to run the lmms-eval on llava-hf/llava-v1.6-x and receive an error:

  File "/data/long_phan/short-circuiting/lmms-eval/lmms_eval/models/llava_hf.py", line 310, in generate_until
    text_outputs = self.tokenizer.batch_decode(cont, skip_special_tokens=True)[0]
IndexError: list index out of range

tracing back I think it is because the llava-hf/llava-v1.6 use LlavaNextForConditionalGeneration and LlavaNextProcessor which does not have image_sizes

ValueError: The following `model_kwargs` are not used by the model: ['image_sizes'] (note: typos in the generate arguments will also show up in this list)

Here is the command that I used:

accelerate launch -m lmms_eval --model llava_hf --model_args pretrained=llava-hf/llava-v1.6-mistral-7b-hf  --tasks llava_in_the_wild --batch_size 1 --output_path ./logs/ #
kcz358 commented 1 month ago

Hi @justinphan3110cais , I have fixed the error. You can pull the main again and try the same command.

justinphan3110cais commented 1 month ago

@kcz358 , Thanks, I saw your fix. However, I think a more robust fix is we can check for the architectures in the AutoConfig? For example current fix will fail if I have my local version of the llava-hf-1.6 or some model trained from llava-hf-1.6?

kcz358 commented 1 month ago

I think both are kind of a hard code fix but maybe checking architectures are more flexible. For now I think you just need to make sure your pretrained path contains 1.6 or 1.5 to separate different versions of llava-hf. And I think the way I parse the results also have some problem. Maybe I will try to address these issues later but if you already have your own fix on it you are welcome to raise a PR.