search

opendatalab / PDF-Extract-Kit

A Comprehensive Toolkit for High-Quality PDF Content Extraction
https://pdf-extract-kit.readthedocs.io/zh-cn/latest/index.html
GNU Affero General Public License v3.0
5.27k stars 357 forks source link

Model and pre-trained model parameters do not match!!! #116

Closed MtrsChJG closed 1 month ago

MtrsChJG commented 1 month ago

I am a green hand using LLM, and the error message shows that the parameter shapes of my model and the pre-trained model do not match. How can I modify the code? I directly used the author's code without any changes. Here are some detailed information:

  1. Python version is 3.12 (Ubuntu 22.04)
  2. Pytorch version 2.3.0
  3. Cuda version is 12.1

Here is the debugging information: 2024-09-06 09:08:25 Started! CustomVisionEncoderDecoderModel init VariableUnimerNetModel init VariableUnimerNetPatchEmbeddings init VariableUnimerNetModel init VariableUnimerNetPatchEmbeddings init CustomMBartForCausalLM init CustomMBartDecoder init Traceback (most recent call last): File "/root/pipeline/PDF-Extract-Kit-main/pdf_extract.py", line 101, in mfr_model, mfr_vis_processors = mfr_model_init(model_configs['model_args']['mfr_weight'], device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/pipeline/PDF-Extract-Kit-main/pdf_extract.py", line 43, in mfr_model_init model = task.build_model(cfg) ^^^^^^^^^^^^^^^^^^^^^ File "/root/pipeline/lib/python3.12/site-packages/unimernet/tasks/base_task.py", line 33, in build_model return model_cls.from_config(model_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/pipeline/lib/python3.12/site-packages/unimernet/models/unimernet/unimernet.py", line 108, in from_config model.load_checkpoint_from_config(cfg) File "/root/pipeline/lib/python3.12/site-packages/unimernet/models/base_model.py", line 97, in load_checkpoint_from_config self.load_from_pretrained(url_or_filename=pretrain_path, **kwargs) File "/root/pipeline/lib/python3.12/site-packages/unimernet/models/blip2_models/blip2.py", line 102, in load_from_pretrained msg = self.load_state_dict(state_dict, strict=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/pipeline/lib/python3.12/site-packages/torch/nn/modules/module.py", line 2189, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for UniMERModel: size mismatch for model.model.decoder.model.decoder.layers.0.self_attn.k_proj.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in c is torch.Size([512, 1024]). size mismatch for model.model.decoder.model.decoder.layers.0.self_attn.k_proj.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current mh.Size([512]).

lllfx commented 1 month ago

I encountered the same problem

hjbiao09 commented 1 month ago

The config does not support the latest version(0.2.0) unimernet. So use pip install unimernet==0.1.6 command to downgrade, and this will solve your issue.

MtrsChJG commented 1 month ago

The config does not support the latest version(0.2.0) unimernet. So use pip install unimernet==0.1.6 command to downgrade, and this will solve your issue.

As you said, it's an issue with the unimernet library version. Thank you very much, my problem has been resolved very well.