ValueError: Image features and image tokens do not match: tokens

[WangRongsheng]

Reminder

• [X] I have read the README and searched the existing issues.

System Info

• llamafactory version: 0.9.1.dev0
• Platform: Linux-4.19.90-2107.6.0.0192.8.oe1.bclinux.x86_64-x86_64-with-glibc2.35
• Python version: 3.10.8
• PyTorch version: 2.1.2+cu118 (GPU)
• Transformers version: 4.46.2
• Datasets version: 3.1.0
• Accelerate version: 1.0.1
• PEFT version: 0.12.0
• TRL version: 0.9.6
• GPU type: NVIDIA A800 80GB PCIe

Reproduction

llamafactory-cli train \ --stage sft \ --do_train True \ --model_name_or_path /root/autodl-tmp/Qwen/Qwen2-VL-2B-Instruct \ --preprocessing_num_workers 16 \ --finetuning_type full \ --template qwen2_vl \ --flash_attn fa2 \ --dataset_dir data \ --dataset pubmed \ --cutoff_len 1024 \ --learning_rate 2e-05 \ --num_train_epochs 1.0 \ --max_samples 10000000 \ --per_device_train_batch_size 2 \ --gradient_accumulation_steps 8 \ --lr_scheduler_type cosine \ --max_grad_norm 1.0 \ --logging_steps 100 \ --save_steps 5000 \ --warmup_steps 0 \ --packing False \ --report_to none \ --output_dir saves/Qwen2-VL-2B-Instruct/full/train_qwen2-vl-2b-1 \ --bf16 True \ --plot_loss True \ --ddp_timeout 180000000 \ --include_num_input_tokens_seen True \ --optim adamw_torch

Expected behavior

No response

Others

No response

[WangRongsheng]

Traceback (most recent call last): File "/root/miniconda3/bin/llamafactory-cli", line 8, in sys.exit(main()) File "/root/autodl-tmp/LLaMA-Factory/src/llamafactory/cli.py", line 111, in main run_exp() File "/root/autodl-tmp/LLaMA-Factory/src/llamafactory/train/tuner.py", line 50, in run_exp run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks) File "/root/autodl-tmp/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 96, in run_sft train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint) File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2123, in train return inner_training_loop( File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2481, in _inner_training_loop tr_loss_step = self.training_step(model, inputs, num_items_in_batch) File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 3579, in training_step loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch) File "/root/autodl-tmp/LLaMA-Factory/src/llamafactory/train/sft/trainer.py", line 88, in compute_loss loss = super().compute_loss(model, inputs, return_outputs, kwargs) File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 3633, in compute_loss outputs = model(inputs) File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/root/miniconda3/lib/python3.10/site-packages/accelerate/utils/operations.py", line 820, in forward return model_forward(args, kwargs) File "/root/miniconda3/lib/python3.10/site-packages/accelerate/utils/operations.py", line 808, in call return convert_to_fp32(self.model_forward(*args, *kwargs)) File "/root/miniconda3/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast return func(args, **kwargs) File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1690, in forward raise ValueError( ValueError: Image features and image tokens do not match: tokens: 1140, features 1350

[hiyouga]

increase cutoff_len