deepspeed zero3 出现 training_eval_loss 图为空白

[ycjcl868]

Reminder

• [X] I have read the README and searched the existing issues.

System Info

A100_SXM_80GB * 8

• llamafactory version: 0.8.3.dev0
• Platform: Linux-5.10.135-x86_64-with-glibc2.31
• Python version: 3.10.14
• PyTorch version: 2.3.0+cu121
• Transformers version: 4.41.2
• Datasets version: 2.18.0
• Accelerate version: 0.30.1
• PEFT version: 0.11.1
• TRL version: 0.8.6
• DeepSpeed version: 0.14.4
• Bitsandbytes version: 0.43.1

Reproduction

llamafactory-cli train --model_name_or_path llama3 \
  --stage sft \ 
  --do_train true \ 
  --finetuning_type lora \ 
  --lora_target all \
  --deepspeed examples/deepspeed/ds_z3_config.json \ 
  --dataset xxxxx \
  --template llama3 \
  --cutoff_len 2048 \ 
  --max_samples 1000 \ 
  --overwrite_cache true \ 
  --preprocessing_num_workers 16 \ 
  --output_dir saves/xxx \
  --logging_steps 10 \ 
  --save_steps 500 \ 
  --plot_loss true \ 
  --overwrite_output_dir true \ 
  --per_device_train_batch_size 1 \ 
  --gradient_accumulation_steps 2 \ 
  --learning_rate 1.0e-5 \ 
  --num_train_epochs 3.0 \ 
  --lr_scheduler_type cosine \
  --warmup_ratio 0.1 \ 
  --fp16 true \ 
  --ddp_timeout 180000000 \ 
  --val_size 0.1 \ 
  --per_device_eval_batch_size 1 \ 
  --eval_strategy steps \ 
  --eval_steps 500

training_eval_loss:

而 eval_results.json 里是有值的

Expected behavior

有曲线

Others

No response

[hiyouga]

减少 eval steps