search

DLYuanGod / TinyGPT-V

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
BSD 3-Clause "New" or "Revised" License
1.24k stars 76 forks source link

Target modules {'query_key_value', 'dense'} not found in the base model #9

Closed FourWinds021 closed 9 months ago

FourWinds021 commented 9 months ago

Hi, I have try to run the demo with the script: python demo_v2.py --cfg-path eval_configs/tinygptv_stage4_eval.yaml --gpu-id 0 but I get the error:

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Initializing Chat Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. You are using a model of type phi to instantiate a model of type phi-msft. This is not supported for all configurations of models and can yield errors. Some weights of the model checkpoint at /user_home/TinyGPT-V-main/phi-2 were not used when initializing PhiForCausalLM: ['model.layers.23.mlp.fc2.weight', 'model.layers.0.mlp.fc2.bias',...] This IS expected if you are initializing PhiForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). This IS NOT expected if you are initializing PhiForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Traceback (most recent call last): File "demo_v2.py", line 63, in model = model_cls.from_config(model_config).to(device) File "/user_home/TinyGPT-V-main/minigpt4/models/minigpt_v2.py", line 170, in from_config model = cls( File "/user_home/TinyGPT-V-main/minigpt4/models/minigpt_v2.py", line 46, in init super().init( File "/user_home/TinyGPT-V-main/minigpt4/models/minigpt_base.py", line 41, in init self.llama_model, self.llama_tokenizer = self.init_llm( File "/user_home/TinyGPT-V-main/minigpt4/models/base_model.py", line 200, in init_llm llama_model = get_peft_model(llama_model, loraconfig) File "/opt/conda/lib/python3.8/site-packages/peft/mapping.py", line 133, in get_peft_model return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config, adapter_name=adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/peft_model.py", line 1043, in init super().init(model, peft_config, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/peft_model.py", line 125, in init self.base_model = cls(model, {adapter_name: peft_config}, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/model.py", line 111, in init super().init(model, config, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/tuners_utils.py", line 90, in init self.inject_adapter(self.model, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/tuners_utils.py", line 250, in inject_adapter raise ValueError( ValueError: Target modules {'query_key_value', 'dense'} not found in the base model. Please check the target modules and try again.

Could you help to solve it? Thank you!

DLYuanGod commented 9 months ago

Hello and thank you for your interest in what we do!

I have a feeling that it is one of the following problems: 1.

  1. I feel that you downloaded the phi model from the wrong address, the model that Microsoft posted on Hugging face is not able to do multimodal training, you need to download the Phi-2 model from susnato:https://huggingface.co/susnato/phi-2. Please check section 2 of Getting Started in the README.md file.

  2. Have you not replaced our Phi_modeling.py file, Please check section 4 of Getting Started in the README.md file.

Please let me know again if it's not one of the above issues.

FourWinds021 commented 9 months ago
  1. I feel that you downloaded the phi model from the wrong address, the model that Microsoft posted on Hugging face is not able to do multimodal training, you need to download the Phi-2 model from susnato:https://huggingface.co/susnato/phi-2. Please check section 2 of Getting Started in the README.md file.
  1. Have you not replaced our Phi_modeling.py file, Please check section 4 of Getting Started in the README.md file.
    • I have copy the Phi_modeling.py and configuration_phi.py to the transformers directory (site-packages/transformers/models/phi/). when not copy the configuration_phi.py file, it get the error: AttributeError: 'PhiConfig' object has no attribute 'n_embd'
DLYuanGod commented 9 months ago

I don't think it's necessary to change all the library files except phi_modeling.py, I think it's a problem with the configuration_phi.py file, or the original transformers library. I have fed back to the second author to help you out, he will contact this issues later.

llizhaoxu commented 9 months ago

Hello

You might have copied the wrong file. The Phi_modeling.py you should copy is TinyGPT-V/modeling_phi.py, instead of TinyGPT-V/minigpt4/models/modeling_phi.py, Could you check whether the file is correct?

Please let me know again if it's not working.

FourWinds021 commented 9 months ago

Hello

You might have copied the wrong file. The Phi_modeling.py you should copy is TinyGPT-V/modeling_phi.py, instead of TinyGPT-V/minigpt4/models/modeling_phi.py, Could you check whether the file is correct?

Please let me know again if it's not working.

Thank you very much, I copy the wrong Phi_modeling.py as you say.