How to increase frame from 8 to a bigger number

[tarunmis]

I need to increase the number of frames for video processing for finetuning and inferencing. Currently it is 8 I want around 32. What is the best way to do it?

[ParkerLischwe]

Also curious about this. @LinB203, is it possible to increase the number of frames used in a video when using LoRA fine tuning? If so, do I just need to change num_frames in processing_video.py to increase amount of frames used for fine tuning as well as inference? What are the differences I should expect when trying to expand the frame amount for fine tuning vs at inference time?

[weikaih04]

Same, wanna do 16 frames in inferencing

[Fritskee]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72

Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101

You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit.

From there on you can continue to select which frames you'd want to provide as input.

[sunyclj]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72

Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101

You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit.

From there on you can continue to select which frames you'd want to provide as input.

直接修改/LanguageBind_Video_merge/config.json中的num_frames报错呢？ 报错如下：size mismatch for vision_model.encoder.layers.23.temporal_embedding: copying a param with shape torch.Size([1, 8, 1024]) from checkpoint, the shape in current model is torch.Size([1, 16, 1024]).

[sunyclj]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72

Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101

You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit.

From there on you can continue to select which frames you'd want to provide as input.

一段视频按照转场分段，每段进行推理描述，最终输出多段的推理结果，请问这个逻辑在哪里增加比较恰当呢？

[Fritskee]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72 Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101 You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit. From there on you can continue to select which frames you'd want to provide as input.

一段视频按照转场分段，每段进行推理描述，最终输出多段的推理结果，请问这个逻辑在哪里增加比较恰当呢？

If you spoke English, I'd be better able to help you. It's also not my first language, so it's a good common ground for us. Anyway, from the little bit that I do understand I'll give you a pointer.

size mismatch for vision_model.encoder.layers.23.temporal_embedding: copying a param with shape torch.Size([1, 8, 1024]) from checkpoint, the shape in current model is torch.Size([1, 16, 1024])

What the above error is saying is that the dimension of your video encoder has a size mismatch. This is because the original video encoder is trained on 8 frames, hence it expects a Tensor of 1, 8, 1024. You now want to sample 16 frames instead of 8 frames. This means your tensor (and therefore your model) should accept this shape. In this case, it probably means you first need to adapt the architecture and then perform a retraining of the vision encoder.

[ybsu]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72 Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101 You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit. From there on you can continue to select which frames you'd want to provide as input.

直接修改/LanguageBind_Video_merge/config.json中的num_frames报错呢？ 报错如下：size mismatch for vision_model.encoder.layers.23.temporal_embedding: copying a param with shape torch.Size([1, 8, 1024]) from checkpoint, the shape in current model is torch.Size([1, 16, 1024]).

I met the same question, how to solve it ? Thanks.

[Fritskee]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72 Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101 You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit. From there on you can continue to select which frames you'd want to provide as input.

直接修改/LanguageBind_Video_merge/config.json中的num_frames报错呢？ 报错如下：size mismatch for vision_model.encoder.layers.23.temporal_embedding: copying a param with shape torch.Size([1, 8, 1024]) from checkpoint, the shape in current model is torch.Size([1, 16, 1024]).

I met the same question, how to solve it ? Thanks.

To train a new vision encoder with the number of frames that you would like to use. In your case I see you are testing with 16 frames, but the vision encoder expects an input tensor of 8 frames. Thus, what you need to do is train a vision encoder on 16 frames, such that the input tensor to the vision encoder is torch.Size([1, 16, 1024]) like your error message says

[ybsu]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72 Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101 You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit. From there on you can continue to select which frames you'd want to provide as input.

直接修改/LanguageBind_Video_merge/config.json中的num_frames报错呢？ 报错如下：size mismatch for vision_model.encoder.layers.23.temporal_embedding: copying a param with shape torch.Size([1, 8, 1024]) from checkpoint, the shape in current model is torch.Size([1, 16, 1024]).

I met the same question, how to solve it ? Thanks.

To train a new vision encoder with the number of frames that you would like to use. In your case I see you are testing with 16 frames, but the vision encoder expects an input tensor of 8 frames. Thus, what you need to do is train a vision encoder on 16 frames, such that the input tensor to the vision encoder is torch.Size([1, 16, 1024]) like your error message says

Thanks, it is a good idea. I wander if it work when I ignore the mismatches parameters, so I add ignore_mismatches_sizes=True in the from_pretrained method; However, the loss is 0 all the time, why it happens?

[Fritskee]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72 Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101 You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit. From there on you can continue to select which frames you'd want to provide as input.

直接修改/LanguageBind_Video_merge/config.json中的num_frames报错呢？ 报错如下：size mismatch for vision_model.encoder.layers.23.temporal_embedding: copying a param with shape torch.Size([1, 8, 1024]) from checkpoint, the shape in current model is torch.Size([1, 16, 1024]).

I met the same question, how to solve it ? Thanks.

To train a new vision encoder with the number of frames that you would like to use. In your case I see you are testing with 16 frames, but the vision encoder expects an input tensor of 8 frames. Thus, what you need to do is train a vision encoder on 16 frames, such that the input tensor to the vision encoder is torch.Size([1, 16, 1024]) like your error message says

Thanks, it is a good idea. I wander if it work when I ignore the mismatches parameters, so I add ignore_mismatches_sizes=True in the from_pretrained method; However, the loss is 0 all the time, why it happens?

You're trying extinguish a fire with a hammer. It doesn't work.

The model isn't trained for this, thus doesn't know how to handle the "extra" parameters you're feeding into it. That's why its all 0.

There is no easy way out here. You need to retrain. It's the only way.