Closed FireAngelx closed 1 month ago
之前发了个issue直接关掉了,没懂这块为啥结果突然变成0.068。我感觉圈的挺明白了,最后的training_loss.png输出没问题,不懂为啥日志会突然变大。。 是不是上面的0.0218才是train_loss呢
model_name_or_path: /home/LAB/liyx24/repository/CodeLlama-13b-Instruct-hf
stage: sft do_train: true finetuning_type: lora lora_target: all
dataset: summer_v3 template: llama2 cutoff_len: 1024 max_samples: 1000000 overwrite_cache: true preprocessing_num_workers: 16
output_dir: saves/CodeLlama-13b-Instruct-hf/lora/train_2024-7-12_v6 logging_steps: 1000 save_steps: 20000 plot_loss: true overwrite_output_dir: true
per_device_train_batch_size: 1 gradient_accumulation_steps: 4 learning_rate: 1.0e-4 num_train_epochs: 5.0 lr_scheduler_type: cosine warmup_ratio: 0.1 lora_rank: 4 fp16: true ddp_timeout: 180000000
val_size: 0.01 per_device_eval_batch_size: 1 eval_strategy: steps eval_steps: 500
使用lora微调CodeLLaMA-13B 配置V100 32GB * 4
No response
希望大佬能帮忙解释一下,如果关掉请说明关掉的原因,之前v0.6没有出现过这个问题,但是v0.8出现了
Reminder
System Info
之前发了个issue直接关掉了,没懂这块为啥结果突然变成0.068。我感觉圈的挺明白了,最后的training_loss.png输出没问题,不懂为啥日志会突然变大。。
是不是上面的0.0218才是train_loss呢
Reproduction
model
model_name_or_path: /home/LAB/liyx24/repository/CodeLlama-13b-Instruct-hf
method
stage: sft do_train: true finetuning_type: lora lora_target: all
dataset
dataset: summer_v3 template: llama2 cutoff_len: 1024 max_samples: 1000000 overwrite_cache: true preprocessing_num_workers: 16
output
output_dir: saves/CodeLlama-13b-Instruct-hf/lora/train_2024-7-12_v6 logging_steps: 1000 save_steps: 20000 plot_loss: true overwrite_output_dir: true
train
per_device_train_batch_size: 1 gradient_accumulation_steps: 4 learning_rate: 1.0e-4 num_train_epochs: 5.0 lr_scheduler_type: cosine warmup_ratio: 0.1 lora_rank: 4 fp16: true ddp_timeout: 180000000
eval
val_size: 0.01 per_device_eval_batch_size: 1 eval_strategy: steps eval_steps: 500
Expected behavior
使用lora微调CodeLLaMA-13B 配置V100 32GB * 4
Others
No response