search

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support 160+ VLMs, 50+ benchmarks
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Apache License 2.0
1.4k stars 195 forks source link

Ovis1.5-Llama3-8B在Hallusion Bench上的指标和榜单上的指标差距过大 #595

Open LIRENDA621 opened 1 week ago

LIRENDA621 commented 1 week ago

1、OpenCompass排行榜的指标是45，但是我们本地测试只有41.30 2、这个差距不是由评判模型造成的。因为需要评判模型处理的'unknown'预测只有14个问题，而这14个问题本身就不是Yes/No问题，我参考了官方给出的预测结果，这14个问题同样回答错误。

image
kennymckormick commented 1 week ago

Hi, @LIRENDA621 , I have re-evaluated this model (torch2.4+cu121, transformers==4.46.2), and got an accuracy of ~42.3%, which looks inferior to previous evaluation results. However, we are not sure whether it's due to randomness.

image

We will re-evaluate this model soon to see if all evaluation results are significant different. If so, we will update the leaderboard and OpenVLMRecords. You can also find the prediction files corresponding to the 45% average accuracy in https://huggingface.co/datasets/VLMEval/OpenVLMRecords and check if there is some problems.