search

huggingface / trl

Train transformer language models with reinforcement learning.
http://hf.co/docs/trl
Apache License 2.0
9.92k stars 1.25k forks source link

Is there a way to finetune a finetuned model? #1937

Closed sherlcok314159 closed 3 weeks ago

sherlcok314159 commented 2 months ago

I use peft and SFTrainer to train a lora model. And I want to fintune this lora model on new datasets. How can I manage this? When using the code below, the grad norm is always zero.

peft_model = ...
model = AutoModelForCausalLM.from_pretrained(
    peft_model,
    device_map='auto',
)
training_args = TrainingArguments(...)
trainer = SFTrainer(...)
trainer.train()
AIR-hl commented 2 months ago

I use peft and SFTrainer to train a lora model. And I want to fintune this lora model on new datasets. How can I manage this? When using the code below, the grad norm is always zero.

peft_model = ...
model = AutoModelForCausalLM.from_pretrained(
    peft_model,
    device_map='auto',
)
training_args = TrainingArguments(...)
trainer = SFTrainer(...)
trainer.train()

you can merge the peft model through merge_and_unload function of peft firstly, and then use the merged model to fine-tune.