Questions about finetuning

[whyiug]

Hi guys, i have some questions about finetuning.

1. In finetune_lora.sh, q and k are selected for lora_target_modules. My understanding is that, considering the efficiency of LoRA, q and v should be chosen instead of q and k. Generally, q represents how to select information (similarity), and v represents the weight of the information.
2. How much does setting tune_vision to true or false affect lora finetuning, please give us some advice.

[qyc-98]

2. LORA's get_peft_model automatically sets the requires_grad for parts of the model excluding LORA to false, thus preventing them from participating in training. Enabling tune_vision would allow both the resampler and VPM to be trained, which may be beneficial for your fine-tuning. We will update the code to default to training the resampler, and through experimentation, we recommend training the resampler by default for most scenarios. As for the VPM part, you can adjust it according to your needs.

[whyiug]

2. LORA's get_peft_model automatically sets the requires_grad for parts of the model excluding LORA to false, thus preventing them from participating in training. Enabling tune_vision would allow both the resampler and VPM to be trained, which may be beneficial for your fine-tuning. We will update the code to default to training the resampler, and through experimentation, we recommend training the resampler by default for most scenarios. As for the VPM part, you can adjust it according to your needs.LORA的get_peft_model会自动将模型中除LORA之外的部分的requires_grad设置为false，从而阻止它们参与训练。启用tune_vision将允许重新采样器和VPM接受训练，这可能有利于您的微调。我们将更新代码以默认训练重采样器，并且通过实验，我们建议在大多数场景下默认训练重采样器。至于VPM部分，你可以根据自己的需要进行调整。

@qyc-98 Thanks for sharing the results of your experiment! So, until you guys update the default code, I just need to comment out this line(https://github.com/OpenBMB/MiniCPM-V/blob/f43e35adcb564b48a1fbbd7989d8ad4b0027f11e/finetune/finetune.py#L264C1-L265C1), right?

[qyc-98]

hi, we update the LORA finetuning code for errors occur during the checkpoint saving process. Please try our code. Thanks!