为什么输出过程中的损失和结果矩阵的输出不一样

Reminder

[X] I have read the README and searched the existing issues.

System Info

之前发了个issue直接关掉了，没懂这块为啥结果突然变成0.068。我感觉圈的挺明白了，最后的training_loss.png输出没问题，不懂为啥日志会突然变大。。是不是上面的0.0218才是train_loss呢

Reproduction

model

model_name_or_path: /home/LAB/liyx24/repository/CodeLlama-13b-Instruct-hf

method

stage: sft do_train: true finetuning_type: lora lora_target: all

dataset

dataset: summer_v3 template: llama2 cutoff_len: 1024 max_samples: 1000000 overwrite_cache: true preprocessing_num_workers: 16

output

output_dir: saves/CodeLlama-13b-Instruct-hf/lora/train_2024-7-12_v6 logging_steps: 1000 save_steps: 20000 plot_loss: true overwrite_output_dir: true

train

per_device_train_batch_size: 1 gradient_accumulation_steps: 4 learning_rate: 1.0e-4 num_train_epochs: 5.0 lr_scheduler_type: cosine warmup_ratio: 0.1 lora_rank: 4 fp16: true ddp_timeout: 180000000

eval

val_size: 0.01 per_device_eval_batch_size: 1 eval_strategy: steps eval_steps: 500

Expected behavior

使用lora微调CodeLLaMA-13B 配置V100 32GB * 4

Others

No response

hiyouga / LLaMA-Factory