search

X-PLUG / mPLUG-Owl

mPLUG-Owl: The Powerful Multi-modal Large Language Model Family
https://www.modelscope.cn/studios/damo/mPLUG-Owl
MIT License
2.25k stars 171 forks source link

What is the process of video input? There is no preprocessing code in your code, but Hugging Face spaces supports video input. #62

Open sdjhshbswp opened 1 year ago

MAGAer13 commented 1 year ago

Hi, the model on huggingface space is the advanced version of mPLUG-Owl which natively support video with temporal related module as input without treating video as multiple frames. The video is tokenized into 65 tokens as image. We will release it very soon.

sdjhshbswp commented 1 year ago

Hi, the model on huggingface space is the advanced version of mPLUG-Owl which natively support video with temporal related module as input without treating video as multiple frames. The video is tokenized into 65 tokens as image. We will release it very soon.

But the paper doesn't mention this temporal-related module. Will this module be detailed in the new version of the paper coming soon?

MAGAer13 commented 1 year ago

Yes, specially we add the local temporal modeling module proposed mPLUG-2, and add a trajectory learnable queries in visual abstractor module. We will include this in the new version of the paper.

feymanwang commented 1 year ago

really need it !

MAGAer13 commented 1 year ago

really need it !

We will release the video version in this week!

feymanwang commented 1 year ago

Awesome! I cant't wait it ! Will you release both code and paper?

MAGAer13 commented 1 year ago

We would not update the paper but we will include the specification of the model's design in the video branch. The code and weight's will be released

feymanwang commented 1 year ago

really need it !

We will release the video version in this week!

Hi, did you release the video version ? I didn't see the updates in this project

MAGAer13 commented 1 year ago

really need it !

We will release the video version in this week!

Hi, did you release the video version ? I didn't see the updates in this project

Sorry for that. The code and weight is under reviewing process, we will release it once the review is done.

LinB203 commented 1 year ago

Hi, this is a wonderful work! I wonder that when will you release the video version ?