Closed louieworth closed 7 months ago
Hi, thanks for bringing this up. I'm not sure if I 100% understand every step you did, but could you try the following:
Since Trainer
registers the parameters for the optimizer only once, at the start, we need to ensure that all parameters that require grads (i.e. the adapter weights for lora 1 and lora 2) are active when we initialize the Trainer
first. So at the very beginning, call model.set_adapter(['lora_1', 'lora_2'])
before initalizing the Trainer
. Next, each time you want to only train one of the two adapters, please activate it using model.set_adapter(['lora_1'])
or model.set_adapter(['lora_2'])
. This should ensure that only the activate adapter gets updated. Did you try this?
Hi, Thanks for your prompt discussion. I have added all parameters to the Trainer with
[mod.requires_grad_(True) for n, mod in my_trainer.model.named_modules() if "lora" in n]
model.set_adapter(['lora_1', 'lora_2'])
However, I think the tricky thing is that the two models are coupled training, i.e., the output from lora_1
is input for lora_2
. And for my best understanding of the source code, the set_adapter
configuration is to set required_grad=True
to its associated adapter but required_grad=False
to others.
So, I can not do this, because whenever I set-adapter('lora_2')
will set the require_grad=False
to the lora_1
model. This may cause some problems. I also tried to explicitly set require_grad=True
but it also raises some errors.
I think the best solution is to load two models rather than two models to avoid non-trivial problems. I will close this problem.
Based on https://huggingface.co/docs/transformers/v4.36.1/en/peft I used to be able to train a multi-adapter model in an interactively way which they are coupled with (same like actor-critic in reinforcement learning):
training.py
The loss function is coupled with
lora_1
andlora_2
so I went toMyTrainer
to override the loss function.However, I met the error in
I followed the suggestion on #1303 from @BenjaminBossan to set
required_grad=True
for all layers with:It can run smoothly without an optimizer bug, but I think it trains the both
lora_1
andlora_2
models. I can also setlora_2
model, and then there is also no optimizer bug. However, I think these two operations can not be implemented to only updatelora_1
model.Followed up
Summary of problem:
self.model.set_adapter("lora_2")
in the last line of the compute loss function, the whole training process runs without bugs. (all layers are in the optimizer states, butlora_1.require_grad=False
andlora_2.require_grad=True
)self.model.set_adapter("lora_1")
, it raises this kind of bug. (all layers are in the optimizer states, butlora_1.require_grad=True
andlora_2.require_grad=False
to reach the goal with only training onlora_1
model).Who can help?
@BenjaminBossan @pacman100 @younesbelkada
Information
Tasks
examples
folderReproduction
I may provide a code sample if it is necessary
Expected behavior
optimization step successful