Open HelloWorldLTY opened 1 year ago
I have the same problem, but I fix it.
It seems you need set loss.requires_grad = True
and it seems normal.
However, the results is not as good. I don't know why.
Maybe the loss here is for basic model rather than LoRA?
Hi, thanks for your exaplaniation. I use another approch to address it.
But my training time is longer.
After using LoRA, it is:
No Lora, it is:
I have no ideas.
Hi, thanks for your exaplaniation. I use another approch to address it.
But my training time is longer.
After using LoRA, it is:
No Lora, it is:
I have no ideas.
Could you share your method to solve this problem? Thanks!
Hi, my approach is to replace all linear module with lora module at first, then try:
lora.mark_only_lora_as_trainable(model)
trainable_params = []
if True:
# if True:
# lora_state_dict = torch.load(model_args.lora_path)
# logger.info(f"Apply LoRA state dict from {model_args.lora_path}.")
# logger.info(lora_state_dict.keys())
# model.load_state_dict(lora_state_dict, strict=False)
trainable_params.append('lora')
if len(trainable_params) > 0:
for name, param in model.named_parameters():
if name.startswith('deberta') or name.startswith('roberta'):
param.requires_grad = False
for trainable_param in trainable_params:
if trainable_param in name:
param.requires_grad = True
break
else:
param.requires_grad = True
Hi, I tried to replace our model's linear layer with lora.linear. However, it seems that all of the components in this module cannot be used for finetuning.
How to address this problem? Thanks a lot.