[BUG] <title>lora单机多卡训练报错：torch.distributed.elastic.multiprocessing.errors.ChildFailedError

[doraemon-plus]

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

• [X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

• [X] 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

torchrun --nproc_per_node 4 --nnodes 1 finetune.py \ --model_name_or_path $MODEL \ --data_path $DATA \ --bf16 True \ --fix_vit True \ --output_dir output_qwen \ --num_train_epochs 5 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 16 \ --evaluation_strategy "no" \ --save_strategy "steps" \ --save_steps 1000 \ --save_total_limit 10 \ --learning_rate 1e-5 \ --weight_decay 0.1 \ --adam_beta2 0.95 \ --warmup_ratio 0.01 \ --lr_scheduler_type "cosine" \ --logging_steps 10 \ --report_to "none" \ --model_max_length 3000 \ --lazy_preprocess True \ --use_lora \ --gradient_checkpointing \ --deepspeed finetune/ds_config_zero2.json

期望行为 | Expected Behavior

No response

复现方法 | Steps To Reproduce

No response

运行环境 | Environment

- OS:
- Python:3.8
- Transformers:4.32
- PyTorch:2.0
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):11.7

备注 | Anything else?

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local rank: 0 (pid: 2548) of binary: /opt/conda/bin/python3

[ShuaiBai623]

能提供更详细的信息吗，以及huggingface上的模型文件是否同步到最新。

[doraemon-plus]

[peace-zy]

您好，遇到了同样的问题，您解决了吗

[HUOUO7]

是不是model_max_length设置太大了？

[ljjcoder]

@doraemon-plus遇到同样的问题，请问你是如何解决的？

[wudidaxuexue]

请问怎么解决的？遇到了相同的问题

[chuangzhidan]

--fix_vit 改为false就报错，请问是什么原因哈？谢谢