search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with one-click evaluation module - lmms-eval.
https://lmms-lab.framer.ai/
Other
2.11k stars 165 forks source link

Bug on Qwen2-VL #282

Open yusalei opened 2 months ago

yusalei commented 2 months ago

When running the code below python3 -m lmms_eval \ --model=qwen2_vl \ --model_args pretrained="Qwen/Qwen2-VL-2B-Instruct",device_map=cuda \ --tasks=mmstar,chartqa \ --batch_size=1 \ --log_samples \ --log_samples_suffix=qwen2_vl \ --output_path="./logs/"

I have encoutered the bug like

TypeError: Can't instantiate abstract class Qwen2_VL without an implementation for abstract method 'generate_until_multi_round'

orrzohar commented 2 months ago

LongVA is also affected:

TypeError: Can't instantiate abstract class LongVA with abstract method generate_until_multi_round
abzb1 commented 2 months ago

cc. https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/281