search

rese1f / MovieChat

[CVPR 2024] MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
https://rese1f.github.io/MovieChat/
BSD 3-Clause "New" or "Revised" License
534 stars 41 forks source link

Questions about forward function #72

Open CserDu opened 5 months ago

CserDu commented 5 months ago

This is a great work. I have some questions about the forward function in moviechat.py. In the forward function, only the encode_videoQformer_visual function is used, but this function seems to directly merge all the video frames according to the similarity until there are 256 frames left, which seems inconsistent with the description in the paper. forward

Espere-1119-Song commented 5 months ago

Thanks for your question! We don't use function encode_videoQformer_visual and forward for our model. We choose VideoLLaMA as our base model, and some functions are not used. Sorry for causing confusion, and we will delete them in our next version.