The percentage of trainable params is around 80%. Is this a normal phenomenon?

Yuliang-Liu / Monkey

【CVPR 2024 Highlight】Monkey (LMM): Image Resolution and Text Label Are Important Things for Large Multi-modal Models

MIT License

1.77k stars 122 forks source link

The percentage of trainable params is around 80%. Is this a normal phenomenon? #103

Closed ZJULiHongxin closed 1 month ago

ZJULiHongxin commented 2 months ago

Thank you for open-sourcing this great @Yuliang-Liu @MelosY

I tried to finetune Qwen-VL using finetune_ds_debug.sh and got these training info: trainable params: 7848497728 || all params: 9708053824 || trainable%: 80.8452

Besides, args.use_lora is also set as False.

Is this large amount of trainable params normal?

I would appreciate it if anyone could help me.

echo840 commented 1 month ago

It's normal that our trainable parameters include LLM, resampler, and LoRA in the visual encoder.