PKU-YuanGroup / Chat-UniVi

[CVPR 2024 Highlight🔥] Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding
https://arxiv.org/abs/2311.08046
Apache License 2.0
827 stars 44 forks source link

mm_projector.bin | size mismatch error #53

Open Asma-Alkhaldi opened 3 months ago

Asma-Alkhaldi commented 3 months ago

Thank you for this excellent work.

I’m trying to do fine-tuning and I followed the exact steps provided, but I believe there are some missing files in Hugging Face.

The file mm_projector.bin is only available in Chat-UniVi/Chat-UniVi-7B-v1.5-Stage-1

Could you clarify which model this bin file is compatible with? Is it:

I have tried all of them, but none worked. I encountered a size mismatch error: size mismatch for weight: copying a param with shape torch.Size([4096, 1024]) from checkpoint, the shape in current model is torch.Size([0]).

Could you please guide me on how to resolve this issue and proceed with fine-tuning?