search

2U1 / Qwen2-VL-Finetune

An open-source implementaion for fine-tuning Qwen2-VL series by Alibaba Cloud.
Apache License 2.0
122 stars 13 forks source link

How does LoRA filter out the torch.nn.modules.Linear parameters in ViT? #13

Open weilanzhikong opened 3 weeks ago

weilanzhikong commented 3 weeks ago

image

I find there linear parameters in ViT

image

2U1 commented 3 weeks ago

https://github.com/2U1/Qwen2-VL-Finetune/blob/80b59b85b5f276cf8759a021eed2f1f895dbbcc1/src/training/train.py#L81-L82

This filters out the vision module linears. If you are not using the vision_lora.