[Question] Question on Accuracy data of SeedBench

[ChimpOnCloud]

Question

Table 1 in this paper showed that the accuracy of SeedBench on vanilla LLaVA-1.5-7B is 60.5. However, when I tried to reproduce this one, I got 66.2 which is higher than that. No response

[mu-cai]

Thanks for your question. We reported the SEED accuracy on Total category, instead of image category.

You can also see this from lmms-eval official results. https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit?gid=0#gid=0

I believe if you run evaluations using lmms-eval for both llava-1.5-7b and M3-7b, you will arrive at the same conclusion.