Loss Nan at the beginning of the training loop

I used the dog face dataset to reproduce the pre-trained model and removed distributed training part in the train_arcface.py, and the loss value is nan. Here are the hyperparameters: Training: 2024-08-30 16:27:10,042-: margin_list (1.0, 0.5, 0.0) Training: 2024-08-30 16:27:10,042-: network r50 Training: 2024-08-30 16:27:10,043-: resume False Training: 2024-08-30 16:27:10,043-: save_all_states False Training: 2024-08-30 16:27:10,043-: output outputs/dog/arcface Training: 2024-08-30 16:27:10,043-: embedding_size 512 Training: 2024-08-30 16:27:10,043-: sample_rate 1.0 Training: 2024-08-30 16:27:10,043-: interclass_filtering_threshold0 Training: 2024-08-30 16:27:10,043-: fp16 False Training: 2024-08-30 16:27:10,043-: batch_size 128 Training: 2024-08-30 16:27:10,043-: optimizer sgd Training: 2024-08-30 16:27:10,044-: lr 0.01 Training: 2024-08-30 16:27:10,044-: momentum 0.9 Training: 2024-08-30 16:27:10,044-: weight_decay 0.0005 Training: 2024-08-30 16:27:10,044-: verbose 2000 Training: 2024-08-30 16:27:10,044-: frequent 10 Training: 2024-08-30 16:27:10,044-: dali False Training: 2024-08-30 16:27:10,044-: gradient_acc 1 Training: 2024-08-30 16:27:10,044-: seed 2048 Training: 2024-08-30 16:27:10,044-: num_workers 8 Training: 2024-08-30 16:27:10,045-: basedir data/PetFace/images Training: 2024-08-30 16:27:10,045-: num_epoch 10 Training: 2024-08-30 16:27:10,045-: warmup_epoch 0 Training: 2024-08-30 16:27:10,045-: train_csv data/PetFace/split/dog/train.csv Training: 2024-08-30 16:27:10,045-: total_batch_size 128 Training: 2024-08-30 16:27:10,045-: warmup_step 0 Training: 2024-08-30 16:27:10,045-: total_step 13150 Or is it possible caused by the class PartialFC in partial_fc.py I modified?

mapooon / PetFace

Loss Nan at the beginning of the training loop #5