search

jayleicn / singularity

[ACL 2023] Official PyTorch code for Singularity model in "Revealing Single Frame Bias for Video-and-Language Learning"
https://arxiv.org/abs/2206.03428
MIT License
129 stars 13 forks source link

MSRVTT-MC Training and Evaluation #5

Closed vateye closed 2 years ago

vateye commented 2 years ago

Hi, I have noticed that MSRVTT-MC evaluation is performed in retrieval manner? I am curious how it be trained? Since there is no training script for training MSRVTT-MC dataset. I think it should be trained in the fashion of training VQA?

jayleicn commented 2 years ago

Please check https://github.com/jayleicn/singularity#evaluation and the relevant part in Section 4.1 of the paper:

For MSRVTT-MC, we follow [31] to use the model trained for the MSRVTT retrieval task, and select the option with the highest retrieval score as the prediction.

vateye commented 2 years ago

Thanks for your patient response. But is there anyway that we can train such multiple choice question in the way of VQA?

jayleicn commented 2 years ago

This repo does not directly support that. But you can make some minor changes to the VQA code to enable it. We would be happy to help if you are interested in making a PR. :)