Qwen2-vl-7B 怎么设置仅微调视觉模型？

modelscope / ms-swift

Use PEFT or Full-parameter to finetune 400+ LLMs or 100+ MLLMs. (LLM: Qwen2.5, Llama3.2, GLM4, Internlm2.5, Yi1.5, Mistral, Baichuan2, DeepSeek, Gemma2, ...; MLLM: Qwen2-VL, Qwen2-Audio, Llama3.2-Vision, Llava, InternVL2, MiniCPM-V-2.6, GLM4v, Xcomposer2.5, Yi-VL, DeepSeek-VL, Phi3.5-Vision, ...)

https://swift.readthedocs.io/zh-cn/latest/Instruction/index.html

Apache License 2.0

4.34k stars 381 forks source link

Qwen2-vl-7B 怎么设置仅微调视觉模型？ #2454

Open thesby opened 1 week ago

Jintao-Huang commented 1 week ago

你是lora还是full

thesby commented 1 week ago

lora 训练的，但是我发现仅用 lora 训练的话，视觉有偶尔有很严重的幻觉，感觉是视觉编码器有问题，想再微调一下视觉模型，请问怎么弄呢？