search

GraphPKU / PiSSA

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
https://arxiv.org/abs/2404.02948
216 stars 8 forks source link

It seems to be an error when getting_peft_model：ValueError: not enough values to unpack (expected 2, got 1) #14

Open Pashisfisuta opened 1 month ago

Pashisfisuta commented 1 month ago

Traceback (most recent call last): File "/ossfs/workspace/sft/sft_all.py", line 161, in train() File "/ossfs/workspace/sft/sft_all.py", line 125, in train Traceback (most recent call last): File "/ossfs/workspace/sft/sft_all.py", line 161, in Traceback (most recent call last): File "/ossfs/workspace/sft/sft_all.py", line 161, in Traceback (most recent call last): File "/ossfs/workspace/sft/sft_all.py", line 161, in model = get_peft_model(model, peft_config) File "/opt/conda/lib/python3.8/site-packages/peft/mapping.py", line 149, in get_peft_model return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config, adapter_name=adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/peft_model.py", line 1395, in init super().init(model, peft_config, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/peft_model.py", line 138, in init self.base_model = cls(model, {adapter_name: peft_config}, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/model.py", line 139, in init super().init(model, config, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/tuners_utils.py", line 166, in init self.inject_adapter(self.model, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/tuners_utils.py", line 372, in inject_adapter self._create_and_replace(peft_config, adapter_name, target, target_name, parent, current_key=key) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/model.py", line 223, in _create_and_replace new_module = self._create_new_module(lora_config, adapter_name, target, kwargs) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/model.py", line 314, in _create_new_module new_module = dispatcher(target, adapter_name, lora_config=lora_config, kwargs) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 1116, in dispatch_default new_module = Linear(target, adapter_name, **kwargs) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 410, in init self.update_layer( File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 117, in update_layer self.pissa_init(adapter_name, init_lora_weights) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 179, in pissa_init Vr, Sr, Ur = svd_lowrank( File "/opt/conda/lib/python3.8/site-packages/torch/_lowrank.py", line 137, in svd_lowrank Traceback (most recent call last): File "/ossfs/workspace/sft/sft_all.py", line 161, in return _svd_lowrank(A, q=q, niter=niter, M=M) File "/opt/conda/lib/python3.8/site-packages/torch/_lowrank.py", line 147, in _svd_lowrank m, n = A.shape[-2:] ValueError: not enough values to unpack (expected 2, got 1)

Pashisfisuta commented 1 month ago

more information: peft==0.11.1 load_model_code :

model = AutoModelForCausalLM.from_pretrained(
model_args.model_name_or_path,
torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(
model_args.model_name_or_path,
use_fast=False,
)
tokenizer.pad_token_id = tokenizer.eos_token_id

peft_config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.1,
    init_lora_weights="pissa_niter_4",
    target_modules=["q_proj", "o_proj", "k_proj", "v_proj", "gate_proj", "up_proj", "down_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)

model.enable_input_require_grads()
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()