LORA微调chatglm3-6b报错：ValueError: Target modules {'v_proj', 'gate_proj', 'k_proj', 'o_proj', 'down_proj', 'q_proj', 'up_proj'} not found in the base model. Please check the target modules and try again. #5368
[X] I have read the README and searched the existing issues.
System Info
ValueError: Target modules {'v_proj', 'gate_proj', 'k_proj', 'o_proj', 'down_proj', 'q_proj', 'up_proj'} not found in the base model. Please check the target modules and try again.
AlexYoung757
commented
2 weeks ago
Reminder
System Info
ValueError: Target modules {'v_proj', 'gate_proj', 'k_proj', 'o_proj', 'down_proj', 'q_proj', 'up_proj'} not found in the base model. Please check the target modules and try again.
Reproduction
[INFO|configuration_utils.py:796] 2024-09-04 21:36:18,532 >> Model config ChatGLMConfig { "_name_or_path": "F:\LargeModel\chatglm3-6b-128k", "add_bias_linear": false, "add_qkv_bias": true, "apply_query_key_layer_scaling": true, "apply_residual_connection_post_layernorm": false, "architectures": [ "ChatGLMModel" ], "attention_dropout": 0.0, "attention_softmax_in_fp32": true, "auto_map": { "AutoConfig": "configuration_chatglm.ChatGLMConfig", "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForCausalLM": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSequenceClassification": "modeling_chatglm.ChatGLMForSequenceClassification" }, "bias_dropout_fusion": true, "classifier_dropout": null, "eos_token_id": 2, "ffn_hidden_size": 13696, "fp32_residual_connection": false, "hidden_dropout": 0.0, "hidden_size": 4096, "kv_channels": 128, "layernorm_epsilon": 1e-05, "model_type": "chatglm", "multi_query_attention": true, "multi_query_group_num": 2, "num_attention_heads": 32, "num_layers": 28, "original_rope": true, "pad_token_id": 0, "padded_vocab_size": 65024, "post_layer_norm": true, "pre_seq_len": null, "prefix_projection": false, "quantization_bit": 0, "rmsnorm": true, "rope_ratio": 500, "seq_length": 131072, "target_modules": [ "q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "down_proj", "up_proj" ], "tie_word_embeddings": false, "torch_dtype": "float16", "transformers_version": "4.41.2", "use_cache": true, "vocab_size": 65024 }
Expected behavior
No response
Others
No response