search

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support 160+ VLMs, 50+ benchmarks
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Apache License 2.0
1.34k stars 188 forks source link

[Model] qwen2vl Added #423

Closed kq-chen closed 2 months ago

luyao-cv commented 2 months ago

你好，我用这份代码，跑出来的指标比github的指标要低。请问是什么原因呢？

模型 docvqa-val chartqa infovqa-val textvqa ocrbench
测试工具 val: VLMEvalKit VLMEvalKit VLMEvalKit VLMEvalKit VLMEvalKit
qwen2 vl 89.262 VLMEvalKit 72.6 VLMEvalKit 64.1 79.2 780
qwen2 vl github 88.34test: 90.1 73.5 65.5 79.7 794

我的环境： cuda 11.8 python: 3.10.14 transformers: 4.45.0.dev0 flash-attn: 2.6.3和2.6.1也试过 pytorch: 2.4.0

xiaoluo333 commented 2 months ago

你好，我用这份代码，跑出来的指标比github的指标要低。请问是什么原因呢？

模型 docvqa-val chartqa infovqa-val textvqa ocrbench 测试工具 val: VLMEvalKit VLMEvalKit VLMEvalKit VLMEvalKit VLMEvalKit qwen2 vl 89.262 VLMEvalKit 72.6 VLMEvalKit 64.1 79.2 780 qwen2 vl github 88.34test: 90.1 73.5 65.5 79.7 794 我的环境： cuda 11.8 python: 3.10.14 transformers: 4.45.0.dev0 flash-attn: 2.6.3和2.6.1也试过 pytorch: 2.4.0

大佬你是怎么跑的？