只全参数微调Qwen2-VL-7B-Instruct的visual.merger部分，冻结其他模型参数，训练过程报错

[wjx-sudo]

Reminder

• [X] I have read the README and searched the existing issues.

System Info

model

model_name_or_path: /Qwen2-VL-7B-Instruct

method

stage: sft do_train: true finetuning_type: full train_mm_proj_only: true #训练多模态投影器 deepspeed: examples/deepspeed/ds_z2_config.json

dataset

dataset: mllm_demo,identity template: qwen2_vl cutoff_len: 1024 max_samples: 1000 overwrite_cache: true

preprocessing_num_workers: 16

output

output_dir: saves/qwen2_vl-7b/full/sft logging_steps: 10 save_steps: 500 plot_loss: true overwrite_output_dir: true

train

per_device_train_batch_size: 1 gradient_accumulation_steps: 2 learning_rate: 1.0e-5 num_train_epochs: 3.0 lr_scheduler_type: cosine warmup_ratio: 0.1 bf16: true ddp_timeout: 180000000

eval

val_size: 0.1 per_device_eval_batch_size: 1 eval_strategy: steps eval_steps: 500

Reproduction

File "/root/anaconda3/envs/llamafactory-w/lib/python3.8/site-packages/accelerate/accelerator.py", line 2143, in backward self.deepspeed_engine_wrapped.backward(loss, kwargs) File "/root/anaconda3/envs/llamafactory-w/lib/python3.8/site-packages/accelerate/utils/deepspeed.py", line 166, in backward self.engine.backward(loss, kwargs) File "/root/anaconda3/envs/llamafactory-w/lib/python3.8/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn ret_val = func(*args, **kwargs) File "/root/anaconda3/envs/llamafactory-w/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1976, in backward self.optimizer.backward(loss, retain_graph=retain_graph) File "/root/anaconda3/envs/llamafactory-w/lib/python3.8/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 2051, in backward self.loss_scaler.backward(loss.float(), retain_graph=retain_graph) File "/root/anaconda3/envs/llamafactory-w/lib/python3.8/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward scaled_loss.backward(retain_graph=retain_graph) File "/root/anaconda3/envs/llamafactory-w/lib/python3.8/site-packages/torch/_tensor.py", line 522, in backward torch.autograd.backward( File "/root/anaconda3/envs/llamafactory-w/lib/python3.8/site-packages/torch/autograd/init.py", line 266, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Expected behavior

No response

Others

No response

[nemonameless]

发现 finetuning_type: full 也不是100%都训

[GoGoZeppeli-towa]

发现 finetuning_type: full 也不是100%都训

或许是freeze_vision_tower默认为true的原因？

[wjx-sudo]

发现 finetuning_type: full 也不是100%都训

确实 我尝试把vision_tower部分参数也加进去，训练过程中会卡住，只能微调llm部分

[nemonameless]

freeze_vision_tower 设置为true发现在自己数据集上训的不如false高

[will-wiki]

@wjx-sudo 同样的问题，非流式训练llm-lora+merger的时候，只训练一个step就卡主了，想问下你解决了吗

[piDack]

坐等好心人解决方案

[Michael4933]

llama-factory对VIT和connector的训练支持似乎确实没做太好，好像就是不支持