纯文本微调GLM-4V报错: ValueError: 151339 is not in list

[skywalkerfmc]

Describe the bug What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程，最好有截图) /home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/deepspeed/runtime/zero/linear.py:47: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead. @autocast_custom_fwd /home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/deepspeed/runtime/zero/linear.py:66: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead. @autocast_custom_bwd Train: 0%| | 0/158 [00:00<?, ?it/s]/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/torch/utils/checkpoint.py:1399: FutureWarning: torch.cpu.amp.autocast(args...) is deprecated. Please use torch.amp.autocast('cpu', args...) instead. with device_autocast_ctx, torch.cpu.amp.autocast(cpu_autocast_kwargs), recompute_context: # type: ignore[attr-defined] Traceback (most recent call last): File "/home/fangmc/code/project/swift/swift/cli/sft.py", line 5, in sft_main() File "/home/fangmc/code/project/swift/swift/utils/run_utils.py", line 32, in x_main result = llm_x(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/code/project/swift/swift/llm/sft.py", line 405, in llm_sft trainer.train(training_args.resume_from_checkpoint) File "/home/fangmc/code/project/swift/swift/trainers/mixin.py", line 538, in train res = super().train(resume_from_checkpoint, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/transformers/trainer.py", line 1948, in train return inner_training_loop( ^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/transformers/trainer.py", line 2289, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/transformers/trainer.py", line 3328, in training_step loss = self.compute_loss(model, inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/code/project/swift/swift/trainers/trainers.py", line 179, in compute_loss outputs = model(inputs) ^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1603, in _call_impl result = forward_call(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/accelerate/utils/operations.py", line 819, in forward return model_forward(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/accelerate/utils/operations.py", line 807, in call return convert_to_fp32(self.model_forward(*args, kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 43, in decorate_autocast return func(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/peft/peft_model.py", line 1430, in forward return self.base_model( ^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 179, in forward return self.model.forward(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/anaconda3/envs/swift/lib/python3.11/site-packages/accelerate/hooks.py", line 169, in new_forward output = module._old_forward(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fangmc/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py", line 1198, in forward boi_token_pos, eoi_token_pos = input_id.index(self.config.boi_token_id), input_id.index( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: 151339 is not in list Train: 0%|

Your hardware and system info Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等) ms-swift版本：2.4.0 GPU: A100 torch版本: 2.4.0

Additional context Add any other context about the problem here(在这里补充其他信息)

[Jintao-Huang]

glm4v的代码是最新的嘛

[skywalkerfmc]

glm4v的代码是最新的嘛

微调的时候自动下载的

[Jintao-Huang]

sh发一下吧，我复现一下

[skywalkerfmc]

sh发一下吧，我复现一下

CUDA_VISIBLE_DEVICES=0,1 swift sft \ --model_type glm4v-9b-chat \ --dataset /home/fangmc/code/project/mlm_project/pure_text.jsonl /home/fangmc/code/project/mlm_project/vqa_data.jsonl \ --max_length 4096 \ --num_train_epochs 2

[Jintao-Huang]

可以复现，应该是glm4v的代码不支持 纯文本的微调，可以给他们提个issue

这里会查找'<|begin_of_image|>', 没找到，然后报错了

[demoninpiano]

好像连纯文本的inference也不支持？

[skywalkerfmc]

好像连纯文本的inference也不支持？

支持的