Loss reaches 0 when finetuning 7B model using 2xA100 80G

[rootally]

I'm using the config below and I load the base model as torch.float16

--model_name_or_path llama_model \ --data_path data.json \ --bf16 True \ --num_train_epochs $3 \ --per_device_train_batch_size 2 \ --per_device_eval_batch_size 2 \ --gradient_accumulation_steps 16 \ --evaluation_strategy "no" \ --save_strategy "steps" \ --save_steps 1200 \ --save_total_limit 3 \ --learning_rate 2e-5 \ --weight_decay 0. \ --warmup_ratio 0.03 \ --lr_scheduler_type "cosine" \ --logging_steps 1 \ --model_max_length 2048 \ --fsdp "full_shard auto_wrap" \ --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \ --tf32 True \ --gradient_checkpointing True \ --lazy_preprocess True \ --report_to tensorboard

[CohenQU]

Hi @rootally, I got the same problem when I finetuning 7B model on 2xA100. Have you fixed it now?