search

mbzuai-oryx / Video-ChatGPT

[ACL 2024 🔥] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.
https://mbzuai-oryx.github.io/Video-ChatGPT
Creative Commons Attribution 4.0 International
1.15k stars 98 forks source link

evaluate_activitynet_qa #115

Open rixejzvdl649 opened 2 months ago

rixejzvdl649 commented 2 months ago

evaluate_activitynet_qa

v_iKclcQEl4zI_10

{'q': 'what is the safety factor of the flip', 'a': 'secondary', 'pred': 'The safety factor of the flip-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-backward-forward-back'}
mmaaz60 commented 2 months ago

Hi @rixejzvdl649,

I appreciate your interest in our work. Please provide some more information on how can we help. Is this the output generated by Video-ChatGPT? Thank You.

YoungjaeDev commented 1 month ago

@mmaaz60 activity_qa accuracy is only showing 15% for me, which is not close to the paper's 35%, what is wrong with the code?

hb-jw commented 1 month ago

Hello, I've also been replicating related benchmarks recently, and these benchmarks are mostly based on GPT-assistant, which seems quite costly. I'd like to ask, approximately how much does each of your evaluations cost?

YoungjaeDev commented 1 month ago

@hb-jw If you're considering the gpt4o-mini, I don't think it's going to cost much. under $10?

hb-jw commented 1 month ago

@hb-jw If you're considering the gpt4o-mini, I don't think it's going to cost much. under $10?

Thank you for your reply! I am using the GPT-3.5 Turbo API, and I have only tested 200 questions from the MSVD-QA in the zero-shot QA setup, which comprises about 1500 samples in total, and it has already cost me $0.90. Based on this calculation, testing the entire MSVD-QA would require $6. However, all the benchmarks (zero-shot QA + videochatgpt benchmark) require 9 similar tests. What are your thoughts on this? Could there be an error in my calculations?