search

BAAI-DCAI / Bunny

A family of lightweight multimodal models.
Apache License 2.0
799 stars 61 forks source link

projector和llm lora训练细节 #57

Closed CanvaChen closed 2 months ago

CanvaChen commented 2 months ago

如果我要继续微调，是否可以冻结projector，只训练llm lora呢？

RussRobin commented 2 months ago

Hi @CanvaChen , thank you for your interest in our work.

Sure you can try freezing projector and train lora only.

There are a lot of ways to sft based on our sft dataset and your own. Since i don't have direct access to your data, and data combination needs a lot of expert experiences, i will list some of the basic ways. You may want to try some of them:

  1. Pretrain on our pretrain dataset and lora finetune on your finetune dataset.
  2. Pretrain on our pretrain dataset, lora finetune on our finetune dataset, and finetune on your own, with the same lora. If you finetune based on our lora CKPT, the codes should not throw any error, but the learning rate scheduler may look weird. You may want to mix your dataset and bunny_695k, and then finetune them together.
  3. Pretrain on our pretrain dataset, lora finetune on bunny_695k, and add a new lora to finetune on your own dataset. The new lora is trainable while projector and the first lora are all frozen. I'm not sure whether this works for you. For implementations, please refer to a previous closed issue: #42 .

In our finetuning, projector and lora are always trainable. Our pretrain weights can be found in the model zoo. Of course you can try freezing projector!

Hope all these things make sense for you! If you still have confusion about implementation, feel free to further comment on this issue. Again, I can't 100% guarantee that such three methods work in your case.

Regards Russell BAAI