StableLM looks GPTNeoX and has query_key_value parameters.
I thought I could apply LoRA to StableLM by specifying target_modules='query_key_value', but I got the following error.
Traceback (most recent call last):
File "/root/workspace/finetune.py", line 288, in <module>
fire.Fire(train)
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/root/workspace/finetune.py", line 189, in train
model = get_peft_model(model, config)
File "/usr/local/lib/python3.10/dist-packages/peft/mapping.py", line 112, in get_peft_model
return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config)
File "/usr/local/lib/python3.10/dist-packages/peft/peft_model.py", line 647, in __init__
super().__init__(model, peft_config, adapter_name)
File "/usr/local/lib/python3.10/dist-packages/peft/peft_model.py", line 91, in __init__
self.base_model = PEFT_TYPE_TO_MODEL_MAPPING[peft_config.peft_type](
File "/usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py", line 132, in __init__
self.add_adapter(adapter_name, self.peft_config[adapter_name])
File "/usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py", line 139, in add_adapter
self._find_and_replace(adapter_name)
File "/usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py", line 225, in _find_and_replace
raise ValueError(
ValueError: Target modules query_key_value not found in the base model. Please check the target modules and try again.
Is there any solution?
Thank you in advance!
I confirmed the base model has `query_key_value` parameters in the attention modules.
```
In [5]: model.named_parameters
Out[5]:
```
StableLM looks GPTNeoX and has
query_key_value
parameters. I thought I could apply LoRA to StableLM by specifyingtarget_modules='query_key_value'
, but I got the following error.Is there any solution? Thank you in advance!
I confirmed the base model has `query_key_value` parameters in the attention modules.
``` In [5]: model.named_parameters Out[5]: