search

tsb0601 / MMVP

260 stars 7 forks source link

How to reproduce the result of Figure 6? #3

Open HenryHZY opened 6 months ago

HenryHZY commented 6 months ago

Hi @tsb0601 @s9xie , thanks for your great projector.

I would like to inquire about the process of reproducing the result of Figure 6 (i.e., MLLM’s performance on visual patterns). In scripts/evaluate_vlm.py, it seems that we have to obtain the logit of image and text.

However, take LLaVA as an example, we cannot directly obtain its logits. Or do you use the input as "\<image> Is it a photo of \<statement>" to LLaVA and obtain the result? I am a little confused about it.

Can you provide an instruction or a script to evaluate LLaVA on MMVP_VLM? With this, I can follow these steps to reproduce the results for other models.

HenryHZY commented 6 months ago

Just find that MMVP and MMVP-VLM share the same images but with different imageID. Do you just obtain the result shown in Figure 4 and then convert the result into Figure 6? As stated in Section 3.3 of the paper, "To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s performance on these patterns." If so, could you please share the imageID mapping file from MMVP to MMVP-VLM (e.g., MMVP 243.jpg -> MMVP-VLM Orientation 2.jpg)?

HenryHZY commented 5 months ago

Any update here? :)

tsb0601 commented 5 months ago

Hi,

Thank you for your interest and questions! Yes, we convert the results from Figure.4 to Figure.6 as the questions in MMVp can also be categorized into visual patterns. We will update the repo tomorrow and add a mapping that contains visual patterns for each question in MMVP.

Sincerely

HenryHZY commented 5 months ago

@tsb0601 Hi, is the release schedule going well?

tsb0601 commented 5 months ago

Hi, sorry for the delay, we will release it this week shortly after CVPR results

yeonju7kim commented 5 months ago

Hi, Thank you for your great work. May I know the mapping of questions and visual patterns?