GaiZhenbiao / Phi3V-Finetuning

Parameter-efficient finetuning script for Phi-3-vision, the strong multimodal language model by Microsoft.
Apache License 2.0
43 stars 11 forks source link

Adding argument for freezing vision backbone. #4

Closed 2U1 closed 2 months ago

2U1 commented 2 months ago

Unfreezing the vision backbone while fine-tuning could be helpful for when finetuning according to the following tech report. https://arxiv.org/pdf/2403.06199

GaiZhenbiao commented 2 months ago

It would be better if freezing the vision backbone were the default option.

2U1 commented 2 months ago

I thought it could be better for the adding --freeze_vision at the train.sh can be easier to see the option for users using this repo.

I've added the freeze_vision option at the train.sh

GaiZhenbiao commented 2 months ago

The lora modules are not added to the vision tower anyway though, unfreezing the vision tower will have no effect.

2U1 commented 2 months ago

Will implement the settings a bit later, thanks