Training YOLOv3 with 1 class, Loss value diverges/explodes

[Robin2091]

I have successfully trained yolov3 tiny with my custom dataset and I got good results. I wish to train yolov3 since it will give me better performance. However, when I train, val_loss becomes lower than the training loss(which didn't happen when I was training tiny) and suddenly for 1 epoch, it became huge e.g. 1900000 and then become small again. After training is done, I get worse results than yolov3 tiny.

Issue #128 suggests that the backbone must be changed which is confusing because it was working with yolov3 tiny.

Training Log for yolov3: Epoch 1/12 1/Unknown - 23s 23s/step - loss: 6968.9478 - yolo_output_0_loss: 186.8290 - yolo_output_1_loss: 744.6382 - yolo_output_2_loss: 6026.9116 - yolo_output_0_accuracy: 0.2855 - yolo_output_1_accuracy: 0.1648 - yolo_output_2_accuracy: 0.1148WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (9.881543). Check your callbacks. 576/Unknown - 310s 537ms/step - loss: 84.7771 - yolo_output_0_loss: 10.7497 - yolo_output_1_loss: 13.8905 - yolo_output_2_loss: 47.7779 - yolo_output_0_accuracy: 0.3479 - yolo_output_1_accuracy: 0.2169 - yolo_output_2_accuracy: 0.1894 Epoch 00001: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_1.tf 576/576 [==============================] - 343s 596ms/step - loss: 84.7771 - yolo_output_0_loss: 10.7497 - yolo_output_1_loss: 13.8905 - yolo_output_2_loss: 47.7779 - yolo_output_0_accuracy: 0.3479 - yolo_output_1_accuracy: 0.2169 - yolo_output_2_accuracy: 0.1894 - val_loss: 0.0000e+00 - val_yolo_output_0_loss: 0.0000e+00 - val_yolo_output_1_loss: 0.0000e+00 - val_yolo_output_2_loss: 0.0000e+00 - val_yolo_output_0_accuracy: 0.0000e+00 - val_yolo_output_1_accuracy: 0.0000e+00 - val_yolo_output_2_accuracy: 0.0000e+00 Epoch 2/12 75/76 [============================>.] - ETA: 0s - loss: 23.6736 - yolo_output_0_loss: 2.7028 - yolo_output_1_loss: 5.4938 - yolo_output_2_loss: 2.4037 - yolo_output_0_accuracy: 0.2863 - yolo_output_1_accuracy: 0.2714 - yolo_output_2_accuracy: 0.2427 Epoch 00002: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_2.tf 576/576 [==============================] - 293s 508ms/step - loss: 28.3737 - yolo_output_0_loss: 8.9154 - yolo_output_1_loss: 4.1589 - yolo_output_2_loss: 2.2028 - yolo_output_0_accuracy: 0.3038 - yolo_output_1_accuracy: 0.3737 - yolo_output_2_accuracy: 0.2008 - val_loss: 22.1670 - val_yolo_output_0_loss: 1.3184 - val_yolo_output_1_loss: 4.9361 - val_yolo_output_2_loss: 1.5103 - val_yolo_output_0_accuracy: 0.2807 - val_yolo_output_1_accuracy: 0.4644 - val_yolo_output_2_accuracy: 0.2628 Epoch 3/12 75/76 [============================>.] - ETA: 0s - loss: 25.1673 - yolo_output_0_loss: 2.0446 - yolo_output_1_loss: 3.0181 - yolo_output_2_loss: 5.6328 - yolo_output_0_accuracy: 0.3268 - yolo_output_1_accuracy: 0.4985 - yolo_output_2_accuracy: 0.2209 Epoch 00003: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_3.tf 576/576 [==============================] - 293s 509ms/step - loss: 24.4548 - yolo_output_0_loss: 1.9194 - yolo_output_1_loss: 4.9885 - yolo_output_2_loss: 2.9414 - yolo_output_0_accuracy: 0.3136 - yolo_output_1_accuracy: 0.3630 - yolo_output_2_accuracy: 0.2768 - val_loss: 19.5821 - val_yolo_output_0_loss: 1.0245 - val_yolo_output_1_loss: 2.3920 - val_yolo_output_2_loss: 1.6807 - val_yolo_output_0_accuracy: 0.0477 - val_yolo_output_1_accuracy: 0.4376 - val_yolo_output_2_accuracy: 0.2763 Epoch 4/12 75/76 [============================>.] - ETA: 0s - loss: 21.0737 - yolo_output_0_loss: 1.4803 - yolo_output_1_loss: 3.4294 - yolo_output_2_loss: 1.7392 - yolo_output_0_accuracy: 0.3902 - yolo_output_1_accuracy: 0.4377 - yolo_output_2_accuracy: 0.2716 Epoch 00004: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_4.tf 576/576 [==============================] - 293s 509ms/step - loss: 20.2237 - yolo_output_0_loss: 1.3930 - yolo_output_1_loss: 2.9907 - yolo_output_2_loss: 1.6747 - yolo_output_0_accuracy: 0.2919 - yolo_output_1_accuracy: 0.3557 - yolo_output_2_accuracy: 0.2483 - val_loss: 18.4381 - val_yolo_output_0_loss: 1.2762 - val_yolo_output_1_loss: 2.3767 - val_yolo_output_2_loss: 1.2190 - val_yolo_output_0_accuracy: 0.2494 - val_yolo_output_1_accuracy: 0.5417 - val_yolo_output_2_accuracy: 0.0157 Epoch 5/12 75/76 [============================>.] - ETA: 0s - loss: 19.0036 - yolo_output_0_loss: 1.3448 - yolo_output_1_loss: 2.8540 - yolo_output_2_loss: 1.3052 - yolo_output_0_accuracy: 0.2948 - yolo_output_1_accuracy: 0.4129 - yolo_output_2_accuracy: 0.1160 Epoch 00005: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_5.tf 576/576 [==============================] - 293s 509ms/step - loss: 20.1187 - yolo_output_0_loss: 1.4060 - yolo_output_1_loss: 3.2820 - yolo_output_2_loss: 2.4093 - yolo_output_0_accuracy: 0.2607 - yolo_output_1_accuracy: 0.4378 - yolo_output_2_accuracy: 0.2370 - val_loss: 19.0202 - val_yolo_output_0_loss: 1.1777 - val_yolo_output_1_loss: 2.9172 - val_yolo_output_2_loss: 2.0744 - val_yolo_output_0_accuracy: 0.2508 - val_yolo_output_1_accuracy: 0.1830 - val_yolo_output_2_accuracy: 0.4283 Epoch 6/12 75/76 [============================>.] - ETA: 0s - loss: 18.7130 - yolo_output_0_loss: 1.5470 - yolo_output_1_loss: 2.8023 - yolo_output_2_loss: 1.5835 - yolo_output_0_accuracy: 0.2242 - yolo_output_1_accuracy: 0.4314 - yolo_output_2_accuracy: 0.4101 Epoch 00006: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_6.tf 576/576 [==============================] - 293s 509ms/step - loss: 17.7897 - yolo_output_0_loss: 1.3377 - yolo_output_1_loss: 2.6806 - yolo_output_2_loss: 1.3562 - yolo_output_0_accuracy: 0.3075 - yolo_output_1_accuracy: 0.4350 - yolo_output_2_accuracy: 0.3706 - val_loss: 16.1347 - val_yolo_output_0_loss: 1.2345 - val_yolo_output_1_loss: 2.0828 - val_yolo_output_2_loss: 1.1010 - val_yolo_output_0_accuracy: 0.3763 - val_yolo_output_1_accuracy: 0.3905 - val_yolo_output_2_accuracy: 0.3180 Epoch 7/12 75/76 [============================>.] - ETA: 0s - loss: 17.4923 - yolo_output_0_loss: 1.3183 - yolo_output_1_loss: 3.2497 - yolo_output_2_loss: 1.2091 - yolo_output_0_accuracy: 0.3735 - yolo_output_1_accuracy: 0.3901 - yolo_output_2_accuracy: 0.4288 Epoch 00007: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_7.tf 576/576 [==============================] - 293s 509ms/step - loss: 17.2040 - yolo_output_0_loss: 1.7552 - yolo_output_1_loss: 2.6324 - yolo_output_2_loss: 1.1916 - yolo_output_0_accuracy: 0.2556 - yolo_output_1_accuracy: 0.3421 - yolo_output_2_accuracy: 0.3569 - val_loss: 15.1383 - val_yolo_output_0_loss: 0.9417 - val_yolo_output_1_loss: 2.0549 - val_yolo_output_2_loss: 1.1000 - val_yolo_output_0_accuracy: 0.5993 - val_yolo_output_1_accuracy: 0.2978 - val_yolo_output_2_accuracy: 0.2487 Epoch 8/12 75/76 [============================>.] - ETA: 0s - loss: 15.5984 - yolo_output_0_loss: 1.2275 - yolo_output_1_loss: 2.4235 - yolo_output_2_loss: 0.9993 - yolo_output_0_accuracy: 0.2670 - yolo_output_1_accuracy: 0.2581 - yolo_output_2_accuracy: 0.4290 Epoch 00008: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_8.tf 576/576 [==============================] - 293s 509ms/step - loss: 15.2628 - yolo_output_0_loss: 1.2862 - yolo_output_1_loss: 2.4007 - yolo_output_2_loss: 1.1596 - yolo_output_0_accuracy: 0.2183 - yolo_output_1_accuracy: 0.3268 - yolo_output_2_accuracy: 0.3042 - val_loss: 18.1469 - val_yolo_output_0_loss: 5.4874 - val_yolo_output_1_loss: 1.9249 - val_yolo_output_2_loss: 1.0867 - val_yolo_output_0_accuracy: 0.3495 - val_yolo_output_1_accuracy: 0.4882 - val_yolo_output_2_accuracy: 0.1583 Epoch 9/12 75/76 [============================>.] - ETA: 0s - loss: 23.1819 - yolo_output_0_loss: 9.6438 - yolo_output_1_loss: 2.4547 - yolo_output_2_loss: 1.2863 - yolo_output_0_accuracy: 0.1718 - yolo_output_1_accuracy: 0.4240 - yolo_output_2_accuracy: 0.1557 Epoch 00009: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_9.tf 576/576 [==============================] - 293s 509ms/step - loss: 17.8527 - yolo_output_0_loss: 3.6907 - yolo_output_1_loss: 2.3600 - yolo_output_2_loss: 1.1514 - yolo_output_0_accuracy: 0.2988 - yolo_output_1_accuracy: 0.3024 - yolo_output_2_accuracy: 0.2733 - val_loss: 191241084.1311 - val_yolo_output_0_loss: 190961328.0000 - val_yolo_output_1_loss: 263738.1875 - val_yolo_output_2_loss: 16000.0000 - val_yolo_output_0_accuracy: 0.1795 - val_yolo_output_1_accuracy: 0.0979 - val_yolo_output_2_accuracy: 0.5285 Epoch 10/12 75/76 [============================>.] - ETA: 0s - loss: 22.2108 - yolo_output_0_loss: 6.4208 - yolo_output_1_loss: 2.6703 - yolo_output_2_loss: 1.1839 - yolo_output_0_accuracy: 0.1459 - yolo_output_1_accuracy: 0.2258 - yolo_output_2_accuracy: 0.1712 Epoch 00010: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_10.tf 576/576 [==============================] - 293s 509ms/step - loss: 23.8357 - yolo_output_0_loss: 6.0317 - yolo_output_1_loss: 2.7543 - yolo_output_2_loss: 1.4570 - yolo_output_0_accuracy: 0.2865 - yolo_output_1_accuracy: 0.3309 - yolo_output_2_accuracy: 0.2202 - val_loss: 921.8003 - val_yolo_output_0_loss: 845.6072 - val_yolo_output_1_loss: 54.8087 - val_yolo_output_2_loss: 6.3111 - val_yolo_output_0_accuracy: 0.3375 - val_yolo_output_1_accuracy: 0.5142 - val_yolo_output_2_accuracy: 0.2554 Epoch 11/12 75/76 [============================>.] - ETA: 0s - loss: 20.9743 - yolo_output_0_loss: 2.0663 - yolo_output_1_loss: 2.8056 - yolo_output_2_loss: 1.2975 - yolo_output_0_accuracy: 0.2277 - yolo_output_1_accuracy: 0.4109 - yolo_output_2_accuracy: 0.1568 Epoch 00011: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_11.tf 576/576 [==============================] - 293s 509ms/step - loss: 19.2025 - yolo_output_0_loss: 1.9748 - yolo_output_1_loss: 2.5246 - yolo_output_2_loss: 1.2540 - yolo_output_0_accuracy: 0.3807 - yolo_output_1_accuracy: 0.4004 - yolo_output_2_accuracy: 0.1473 - val_loss: 15.6889 - val_yolo_output_0_loss: 1.2305 - val_yolo_output_1_loss: 1.9561 - val_yolo_output_2_loss: 1.0615 - val_yolo_output_0_accuracy: 0.0113 - val_yolo_output_1_accuracy: 0.1628 - val_yolo_output_2_accuracy: 0.0702 Epoch 12/12 75/76 [============================>.] - ETA: 0s - loss: 16.2245 - yolo_output_0_loss: 1.4228 - yolo_output_1_loss: 2.4705 - yolo_output_2_loss: 1.0931 - yolo_output_0_accuracy: 0.3623 - yolo_output_1_accuracy: 0.2034 - yolo_output_2_accuracy: 0.0861 Epoch 00012: saving model to /storage/4_2734_2734_8/checkpoint/yolov3_train_12.tf 576/576 [==============================] - 293s 509ms/step - loss: 15.1621 - yolo_output_0_loss: 1.4776 - yolo_output_1_loss: 2.3831 - yolo_output_2_loss: 1.1380 - yolo_output_0_accuracy: 0.3685 - yolo_output_1_accuracy: 0.3166 - yolo_output_2_accuracy: 0.1561 - val_loss: 12.6150 - val_yolo_output_0_loss: 1.0471 - val_yolo_output_1_loss: 1.8891 - val_yolo_output_2_loss: 0.9585 - val_yolo_output_0_accuracy: 0.3480 - val_yolo_output_1_accuracy: 0.2403 - val_yolo_output_2_accuracy: 0.1155

[tsk15535904190]

you final-loss is 15.1621 , I think it's works , and you must set lower threshold