昇腾多卡训练问题

1737686924 commented 3 months ago

Reminder

[X] I have read the README and searched the existing issues.

Reproduction

脚本如下： deepspeed --num_gpus 4 src/train_bash.py \ --deepspeed examples/deepspeed/ds_z3_config.json \ --stage sft \ --do_train \ --model_name_or_path /data/applications/LMD-BF/backend/BaseModels/internlm2-chat-20b-sft/internlm2-chat-20b-sft/ \ --dataset identity \ --template intern2 \ --finetuning_type lora \ --lora_target wqkv \ --output_dir saves/internlm2-chat-20b-sft/lora/sft \ --overwrite_cache true \ --overwrite_output_dir true \ --cutoff_len 1024 \ --preprocessing_num_workers 16 \ --per_device_train_batch_size 2 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 4 \ --lr_scheduler_type cosine \ --logging_steps 10 \ --save_steps 100 \ --eval_steps 2 \ --evaluation_strategy steps \ --load_best_model_at_end \ --learning_rate 5e-5 \ --num_train_epochs 10.0 \ --val_size 0.1 \ --ddp_timeout 180000000 \ --plot_loss \ --fp16 stage:0,1,2,3均报错应是爆显存 2ef85aebbc28dedf3a2b03c28471e72