Xtuner和VLMEvalKit针对MMBench的结果不对齐

[thomas-yanxin]

基于Xtuner训练LLaVA-Phi3

使用Xtuner评估MMBench-Dev-EN：0.7096 使用VLMEvalKit评估MMBench-Dev-EN：0.549828

[LZHgrla]

@thomas-yanxin 请问 vlmevalkit 的 model config 是怎么配置的？

[thomas-yanxin]

xtuner_series = { 'llava-internlm2-7b': partial(LLaVA_XTuner, llm_path='internlm/internlm2-chat-7b', llava_path='xtuner/llava-internlm2-7b', visual_select_layer=-2, prompt_template='internlm2_chat'), 'llava-internlm2-20b': partial(LLaVA_XTuner, llm_path='internlm/internlm2-chat-20b', llava_path='xtuner/llava-internlm2-20b', visual_select_layer=-2, prompt_template='internlm2_chat'), 'llava-internlm-7b': partial(LLaVA_XTuner, llm_path='internlm/internlm-chat-7b', llava_path='xtuner/llava-internlm-7b', visual_select_layer=-2, prompt_template='internlm_chat'), 'llava-v1.5-7b-xtuner': partial(LLaVA_XTuner, llm_path='lmsys/vicuna-7b-v1.5', llava_path='xtuner/llava-v1.5-7b-xtuner', visual_select_layer=-2, prompt_template='vicuna'), 'llava-v1.5-13b-xtuner': partial(LLaVA_XTuner, llm_path='lmsys/vicuna-13b-v1.5', llava_path='xtuner/llava-v1.5-13b-xtuner', visual_select_layer=-2, prompt_template='vicuna'), 'llava-phi-3': partial(LLaVA_XTuner, llm_path='microsoft/Phi-3-mini-4k-instruct', llava_path='xtuner/llava-phi-3-mini-xtuner, visual_select_layer=-2, prompt_template='phi3_chat'), }

[LZHgrla]

@thomas-yanxin 因为llava-phi-3的llm也微调了，所以llm_path不应是原llm

'llava-phi-3': partial(
    LLaVA_XTuner,
-   llm_path='microsoft/Phi-3-mini-4k-instruct',
+   llm_path='xtuner/llava-phi-3-mini-xtuner',
    llava_path='xtuner/llava-phi-3-mini-xtuner',
    visual_select_layer=-2,
    prompt_template='phi3_chat'),

[thomas-yanxin]

solved，tks