Cannot reproduce Zero-shot Video-QA (MSVD)

[dcahn12]

Thanks for your contribution!

I tried to reproduce your result (Zero-shot VideoQA on MSVD dataset) with the given pretrained weights. (EVA-G & LLaVA1.5-VideoChatGPT-Instruct 7B).

But the result is completely different from your paper. (Reproduced result is shown below)

Can you check this?

[yanwei-li]

Hi, it seems a huge gap, could you please give more details? We use the this model and this script for evaluation.

[dcahn12]

I also used the same model (llama-vid-7b-full-224-video-fps-1) and employed the same script to generate responses to relevant questions. However, when evaluating GPT-based evaluations, I simply omitted the "--api_base" argument because it caused an error during the evaluation process. Could you please provide the model-generated answer responses for the MSVD-QA dataset?

[yanwei-li]

Hi, we provide the prediction in pred.json and also GPT3.5 evaluated results in results.json. We also re-evaluate the model, because GPT-based evaluation may have performance bias (give different results at each turn), but still within an acceptable range:

[yanwei-li]

Close it now, please reopen it if you have further questions.

[Felix0805]

Hello，can you tell me where can I download the msvd dataset? Thank you very much.