search

h-zhao1997 / cobra

Cobra: Extending Mamba to Multi-modal Large Language Model for Efficient Inference
MIT License
259 stars 8 forks source link

Error loading model parameters: Error(s) in loading state_dict for FusedMLPProjector #9

Open WenjunHuang94 opened 6 months ago

WenjunHuang94 commented 6 months ago

Error loading model parameters, it should be that the loaded model does not match the previously defined model parameters image

pretrained_checkpoint = '/home/hwj/.cache/huggingface/hub/models--han1997--cobra/snapshots/3d1aa9101b8276f9c721237e685cc83ef1d0f79f/cobra+3b/checkpoints/latest-checkpoint.pt'

The model structure of VLM is printed as follows: vlm-model.txt

WenjunHuang94 commented 6 months ago

我对代码做了一些轻微的修改，修改的原因是程序报告了错误。修改如下： image

但我不明白这是否会导致模型创建时结构变化，从而引导模型参数加载时对应不上model结构。

WenjunHuang94 commented 6 months ago

其中 config_json = '/home/hwj/.cache/huggingface/hub/models--han1997--cobra/snapshots/3d1aa9101b8276f9c721237e685cc83ef1d0f79f/cobra+3b/config.json' image

config_json内容如下： config.json

h-zhao1997 commented 6 months ago

@huangwenjunlovedy 请用requirements.txt指定版本的transformers，高版本的transformers目前会有这样的兼容性问题

WenjunHuang94 commented 6 months ago

谢谢。在您刚回复的时候，我刚好简单地修改了下代码跑通了。就是把projector模型的形状，改成和权重参数形状一样。 image 02dd1415e902df40b4b5fe67314dc55 973e2a60619391fdd412892beea08e2

我等会试着按您说的，不自己做代码修改。而是使用requirements.txt指定版本的transformers去尝试跑通！ 再次感谢您的回复！

WenjunHuang94 commented 6 months ago

@huangwenjunlovedy 请用requirements.txt指定版本的transformers，高版本的transformers目前会有这样的兼容性问题

我没有在您的cobra github上找到requirements.txt。你能提供下吗

h-zhao1997 commented 6 months ago

@huangwenjunlovedy Sorry，我想说的是pyproject.toml

WenjunHuang94 commented 6 months ago

@huangwenjunlovedy Sorry，我想说的是pyproject.toml

已解决。十分感谢！ 想请问下后续如果要用高版本的transformers，应该怎么避免这个问题呢