search

DAMO-NLP-SG / VideoLLaMA2

VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
Apache License 2.0
800 stars 55 forks source link

Json files of the MVSD-QA dataset #100

Open Hou9612 opened 3 weeks ago

Hou9612 commented 3 weeks ago

Congratulations on this wonderful work!

The readme file suggests preparing test_q.json and test_a.json for evaluation on the MVSD-QA dataset, but on the official website of the MVSD-QA dataset, I can only download test_qa.json. How can I make test_q.json and test_a.json files based on the test_qa.json file, or where can I get test_q.json and test_a.json files?

clownrat6 commented 6 days ago

VideoChatGPT doesn't provide MSVD official questions and answers, but provide the generation method of questions and answers. Therefore, different papers may adopt different test set of MSVD dataset. Following previous works, the MSVD-QA dataset adopted by us is generated by Video-LLaVA. However, we do not suggest to evaluate VideoLLM on MSVD-QA because it seems to be evaluated unconsistently across different papers.

Hou9612 commented 5 days ago

VideoChatGPT doesn't provide MSVD official questions and answers, but provide the generation method of questions and answers. Therefore, different papers may adopt different test set of MSVD dataset. Following previous works, the MSVD-QA dataset adopted by us is generated by Video-LLaVA. However, we do not suggest to evaluate VideoLLM on MSVD-QA because it seems to be evaluated unconsistently across different papers.

Thanks very much for your reply!