search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.03k stars 53 forks source link

[WIP] adding mmbench dev evaluation (#75) #46

Closed Luodian closed 2 months ago

Luodian commented 2 months ago

LLaVA-v1.5-7B eval results

image image image
Luodian commented 2 months ago

@pufanyi Please help us to test if this PR works for LLaVA-v1.5 and LLaVA-v1.6 (using official repo code) model.

Luodian commented 2 months ago 
python -m accelerate.commands.launch \
    --main_process_port=12566 \
    --num_processes=8 \
    lmms_eval \
    --model=llava \
    --model_args=pretrained=liuhaotian/llava-v1.5-13b,conv_template=vicuna_v1 \
    --tasks=mmbench_en_dev,mmbench_cn_dev,mmbench_cn_cc \
    --batch_size=1 \
    --log_samples \
    --log_samples_suffix=debug \
    --output_path=./logs/ \
    --verbosity=DEBUG