search

jayleicn / singularity

[ACL 2023] Official PyTorch code for Singularity model in "Revealing Single Frame Bias for Video-and-Language Learning"
https://arxiv.org/abs/2206.03428
MIT License
130 stars 14 forks source link

How to run Open-ended QA #16

Closed ShuangLI59 closed 2 years ago

ShuangLI59 commented 2 years ago

Can I ask how to run open-ended QA on ActivityNet? (Generate text answer instead of doing classification)

jayleicn commented 2 years ago

Hi @ShuangLI59, Please check the instructions in https://github.com/jayleicn/singularity#question-answering.

ShuangLI59 commented 2 years ago

Thanks for your reply. Sorry, I couldn't find the code for generating open-language answers. The link you provided is for selecting answers from predefined classes. Given a video and a question, how to generate an open-language answer? Could you provide the command? Thanks.

jayleicn commented 2 years ago

Yes, we perform open-ended QA in a somewhat closed setup, by selecting only among the top-N most frequent answers. That is how training and evaluation (last paragraph) work.

To lose this closed constraint, you may need to modify the inference code here, especially you need to change this function rank_answer.py. You are welcome to make a PR on it.

ShuangLI59 commented 2 years ago

Thanks for your reply. I have one question about the rank_answer function. Can I ask why answer_ids is provided during inference? Do you mind providing the code of experiments in Appendix A.1 / Figure 8a (Open-ended QA model)?