Hi, first of all, thank you for your excellent work on the MMVP Benchmark. It's a great resource.
I've encountered a discrepancy in the answers provided for questions 279 and 280 regarding the position (standing or sitting) of an elderly person in the respective images. As per the Questions.csv file, the answers are listed as follows:
"279,Is the elderly person standing or sitting in the picture?,(a) Standing (b) Sitting,(a)"
"280,Is the elderly person standing or sitting in the picture?,(a) Standing (b) Sitting,(b)"
However, upon reviewing the images linked for these questions, it seems there's a mismatch:
Figure 279:
Figure 280:
Based on these images, it appears the correct answers should be inverted:
For question 279, the elderly person is sitting, so the correct answer should be (b) Sitting.
For question 280, the elderly person is standing, so the correct answer should be (a) Standing.
Given the benchmark consists of only 150 pairs of questions, even a single discrepancy can lead to a significant accuracy difference of approximately 2*0.67%. It would be beneficial for the integrity of the benchmark to correct this inconsistency.
Hi, first of all, thank you for your excellent work on the MMVP Benchmark. It's a great resource.
I've encountered a discrepancy in the answers provided for questions 279 and 280 regarding the position (standing or sitting) of an elderly person in the respective images. As per the Questions.csv file, the answers are listed as follows:
However, upon reviewing the images linked for these questions, it seems there's a mismatch:
Figure 279:
Figure 280:
Based on these images, it appears the correct answers should be inverted:
For question 279, the elderly person is sitting, so the correct answer should be (b) Sitting.
For question 280, the elderly person is standing, so the correct answer should be (a) Standing.
Given the benchmark consists of only 150 pairs of questions, even a single discrepancy can lead to a significant accuracy difference of approximately 2*0.67%. It would be beneficial for the integrity of the benchmark to correct this inconsistency.
Thank you for considering this correction.