search

mbzuai-oryx / Video-ChatGPT

[ACL 2024 🔥] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.
https://mbzuai-oryx.github.io/Video-ChatGPT
Creative Commons Attribution 4.0 International
1.17k stars 102 forks source link

Great work! #16

Closed wyzjack closed 1 year ago

wyzjack commented 1 year ago

Hi, congrats on the great work and very impressive performance!

I have a small question on the Spatio-Temporal features using CLIP. So in the OneDrive downloading path (https://mbzuaiac-my.sharepoint.com/:f:/g/personal/hanoona_bangalath_mbzuai_ac_ae/EnLRDehrr8lGqHpC5w1zZ9QBnsiVffYy5vCv8Hl14deRcg?e=Ul5DUE) you provided, there seems to be no "v_CL6TbOgnLzA.pkl" file, which exists in https://github.com/mbzuai-oryx/Video-ChatGPT/blob/main/docs/train_video_ids.txt, and will cause bug when run training script. Could you help?

I would appreciate it very much if you could reply. Thanks in advance.

wyzjack commented 1 year ago

Hi authors, I would appreciate it very much if you could reply. Thanks!

mmaaz60 commented 1 year ago

Hi @wyzjack,

Thank you for your interest in our work. Some of the video files were corrupted in our case and it could be the reason why the clip feature files are missing, and the reason of the mismatch. You can try skipping these videos as we did in our experiments.

Otherwise, please try using VideoInstruct_Dataset_Train.json from the provided resources which should not cause any mismatch.

Please let me know if it works. Thank you.

wyzjack commented 1 year ago

Great, thanks a lot for your reply and it resolved my problem. Congratulations again on your nice work!

wyzjack commented 1 year ago

Thanks

tkarthikeyan132 commented 11 months ago

Hi @mmaaz60 , I could not find this file VideoInstruct_Dataset_Train.json instead i found this VideoInstruct_Dataset.json (https://mbzuaiac-my.sharepoint.com/:u:/g/personal/hanoona_bangalath_mbzuai_ac_ae/EWxYslvDeX1PijKWM_WxTkkBDXDDD350YnUQOkbcL8V7Xg?e=Lq9itD)

Hi @wyzjack, How did you finally resolve this error

Please elaborate it, I am new to this

zhanwenchen commented 11 months ago

Hi @mmaaz60 , I could not find this file VideoInstruct_Dataset_Train.json instead i found this VideoInstruct_Dataset.json (https://mbzuaiac-my.sharepoint.com/:u:/g/personal/hanoona_bangalath_mbzuai_ac_ae/EWxYslvDeX1PijKWM_WxTkkBDXDDD350YnUQOkbcL8V7Xg?e=Lq9itD)

Hi @wyzjack, How did you finally resolve this error

Please elaborate it, I am new to this

Please download video_chatgpt_training_removed.json produced by @wyzjack. (sha256sum is 2efa20a69ba16f07f6de87097f25dd6176ebd98e71fff6f6280e512f65933484) from my Google Drive: https://drive.google.com/file/d/1Wb0vYuavCoBYos6LXjY5CKfQjU6UqlIi/view?usp=drive_link.