search

david8862 / keras-YOLOv3-model-set

end-to-end YOLOv4/v3/v2 object detection pipeline, implemented on tf.keras with different technologies
MIT License
640 stars 222 forks source link

Training problem using different tensorflow version #153

Open tangyugu opened 3 years ago

tangyugu commented 3 years ago

I face some problems while training using tensorflow 2.2.0 result: Pascal VOC AP evaluation motorbike: AP 0.0000, precision 0.0000, recall 0.0000 mAP@IoU=0.40 result: 0.0000 mPrec@IoU=0.40 result: 0.0000 mRec@IoU=0.40 result: 0.0000

But I use tensorflow 2.0.0 training (using the same data and parameters) Pascal VOC AP evaluation motorbike: AP 0.2224, precision 0.7443, recall 0.2551 mAP@IoU=0.40 result: 22.243578 mPrec@IoU=0.40 result: 74.426230 mRec@IoU=0.40 result: 25.505618

I am not sure what happened why the result is too much different while using different tensorflow version. It seems that some problems happened. Maybe the loss function is incompatible using tf 2.2?

david8862 commented 3 years ago

@tangyugu sounds weired. I've ever used tf 2.2 for training and it works fine. did you meet any error msg under tf 2.2?

tangyugu commented 3 years ago

I try to use tf 2.4, but it seems like the same problem. These are the log showing while training using tf 2.4 and tf 2.0 (For using the same data, I set the seeds for get_dataset function) Print the first 3 epoch (I trained 20 epochs before. Tf 2.0 worked but tf 2.2 ↑ did not worked.)

tf 2.0

root@a6862131cb16:/workspace/tony/keras-YOLOv3-model-set# python train.py 
--model_type=yolo4_darknet 
--anchors_path=anchors/motorbike_yolov4_anchor_csp_onlycoco_416.txt 
--annotation_file=object_dataset/combine_annotation.txt 
--classes_path=classes/motorbike_classes.txt 
--gpu_num=1 
--batch_size=32  
--save_eval_checkpoint 
--optimizer=sgd 
--freeze_level=1 
--model_image_size=416x416 
--eval_online 
--eval_epoch_interval=3

using default seed: 42
2021-02-23 01:38:49.052599: E tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] The TF_FORCE_GPU_ALLOW_GROWTH environment variable is set but could not be parsed: "True". Valid values are "true" or "false". Using original config value of 1.
backbone layers number: 250
Load weights weights/yolov4.h5.
Create  yolo4_darknet model with 9 anchors and 1 classes.
model layer number: 370
Freeze the first 250 layers of total 370 layers.
Transfer training stage
Train on 3152 samples, val on 350 samples, with batch size 32, input_shape (416, 416).
Train for 98 steps, validate for 10 steps
Epoch 1/20
 1/98 [..............................] - ETA: 45:40 - loss: 8926.2432 - location_loss: 2.4413 - confidence_loss: 8921.2695- class_loss: 1.6666WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (3.243871). Check your callbacks.
 2/98 [..............................] - ETA: 29:43 - loss: 4491.7008 - location_loss: 2.6371 - confidence_loss: 4486.3286 3/98 [..............................] - ETA: 20:27 - loss: 3017.6518 - location_loss: 2.8320 - confidence_loss: 3011.9895 4/98 [>.............................] - ETA: 15:49 - loss: 2268.1950 - location_loss: 2.5331 - confidence_loss: 2263.0999 5/98 [>.............................] - ETA: 13:02 - loss: 1826.5883 - location_loss: 2.5553 - confidence_loss: 1821.4467 6/98 [>.............................] - ETA: 11:09 - loss: 1531.6072 - location_loss: 2.6591 - confidence_loss: 1526.2902 7/98 [=>............................] - ETA: 9:49 - loss: 1316.3820 - location_loss: 2.5584 - confidence_loss: 1311.2361 - 8/98 [=>............................] - ETA: 8:48 - loss: 1154.2892 - location_loss: 2.4565 - confidence_loss: 1149.3135 - 9/98 [=>............................] - ETA: 8:01 - loss: 1030.2709 - location_loss: 2.5048 - confidence_loss: 1025.2363 -10/98 [==>...........................] - ETA: 7:23 - loss: 929.3012 - location_loss: 2.4279 - confidence_loss: 924.3865 - c11/98 [==>...........................] - ETA: 6:51 - loss: 847.3573 - location_loss: 2.4252 - confidence_loss: 842.4368 - c12/98 [==>...........................] - ETA: 6:24 - loss: 779.3236 - location_loss: 2.4565 - confidence_loss: 774.3642 - c13/98 [==>...........................] - ETA: 6:01 - loss: 722.2092 - location_loss: 2.5112 - confidence_loss: 717.1811 - c14/98 [===>..........................] - ETA: 5:41 - loss: 672.5981 - location_loss: 2.5140 - confidence_loss: 667.5668 - c15/98 [===>..........................] - ETA: 5:23 - loss: 630.1777 - location_loss: 2.5662 - confidence_loss: 625.0739 - c16/98 [===>..........................] - ETA: 5:08 - loss: 592.2711 - location_loss: 2.5569 - confidence_loss: 587.1859 - c17/98 [====>.........................] - ETA: 4:54 - loss: 559.1785 - location_loss: 2.5818 - confidence_loss: 554.0604 - c18/98 [====>.........................] - ETA: 4:42 - loss: 529.5030 - location_loss: 2.5781 - confidence_loss: 524.3997 - c19/98 [====>.........................] - ETA: 4:31 - loss: 502.8909 - location_loss: 2.5748 - confidence_loss: 497.7931 - c20/98 [=====>........................] - ETA: 4:20 - loss: 478.8945 - location_loss: 2.5714 - confidence_loss: 473.8047 - c21/98 [=====>........................] - ETA: 4:10 - loss: 457.1462 - location_loss: 2.5547 - confidence_loss: 452.0949 - c22/98 [=====>........................] - ETA: 4:02 - loss: 437.4316 - location_loss: 2.5448 - confidence_loss: 432.4011 - c23/98 [======>.......................] - ETA: 3:53 - loss: 419.3147 - location_loss: 2.5161 - confidence_loss: 414.3354 - c24/98 [======>.......................] - ETA: 3:46 - loss: 403.0775 - location_loss: 2.5413 - confidence_loss: 398.0716 - c25/98 [======>.......................] - ETA: 3:39 - loss: 387.6027 - location_loss: 2.5003 - confidence_loss: 382.6693 - c26/98 [======>.......................] - ETA: 3:32 - loss: 373.5721 - location_loss: 2.4988 - confidence_loss: 368.6492 - c27/98 [=======>......................] - ETA: 3:25 - loss: 360.8973 - location_loss: 2.5196 - confidence_loss: 355.9457 - c28/98 [=======>......................] - ETA: 3:19 - loss: 348.6375 - location_loss: 2.4898 - confidence_loss: 343.7381 - c29/98 [=======>......................] - ETA: 3:14 - loss: 337.4346 - location_loss: 2.4979 - confidence_loss: 332.5324 - c30/98 [========>.....................] - ETA: 3:08 - loss: 327.0466 - location_loss: 2.5139 - confidence_loss: 322.1345 - c31/98 [========>.....................] - ETA: 3:03 - loss: 316.9373 - location_loss: 2.4770 - confidence_loss: 312.0900 - c32/98 [========>.....................] - ETA: 2:58 - loss: 307.6401 - location_loss: 2.4611 - confidence_loss: 302.8227 - c33/98 [=========>....................] - ETA: 2:53 - loss: 299.1360 - location_loss: 2.4766 - confidence_loss: 294.2993 - c34/98 [=========>....................] - ETA: 2:48 - loss: 290.7791 - location_loss: 2.4524 - confidence_loss: 285.9897 - c35/98 [=========>....................] - ETA: 2:44 - loss: 282.9239 - location_loss: 2.4300 - confidence_loss: 278.1788 - c36/98 [==========>...................] - ETA: 2:40 - loss: 275.6131 - location_loss: 2.4204 - confidence_loss: 270.8863 - c37/98 [==========>...................] - ETA: 2:36 - loss: 268.5978 - location_loss: 2.4023 - confidence_loss: 263.9113 - c38/98 [==========>...................] - ETA: 2:32 - loss: 261.9998 - location_loss: 2.3937 - confidence_loss: 257.3367 - c39/98 [==========>...................] - ETA: 2:28 - loss: 255.7000 - location_loss: 2.3781 - confidence_loss: 251.0684 - c40/98 [===========>..................] - ETA: 2:24 - loss: 250.0129 - location_loss: 2.3986 - confidence_loss: 245.3566 - c41/98 [===========>..................] - ETA: 2:21 - loss: 244.4750 - location_loss: 2.4092 - confidence_loss: 239.8143 - c42/98 [===========>..................] - ETA: 2:17 - loss: 239.0842 - location_loss: 2.3971 - confidence_loss: 234.4514 - c43/98 [============>.................] - ETA: 2:14 - loss: 233.9668 - location_loss: 2.3944 - confidence_loss: 229.3464 - c44/98 [============>.................] - ETA: 2:10 - loss: 229.1224 - location_loss: 2.3937 - confidence_loss: 224.5069 - c45/98 [============>.................] - ETA: 2:07 - loss: 224.5228 - location_loss: 2.3968 - confidence_loss: 219.9122 - c46/98 [=============>................] - ETA: 2:04 - loss: 220.0095 - location_loss: 2.3865 - confidence_loss: 215.4211 - c47/98 [=============>................] - ETA: 2:00 - loss: 215.8046 - location_loss: 2.3945 - confidence_loss: 211.2147 - c48/98 [=============>................] - ETA: 1:57 - loss: 211.7274 - location_loss: 2.3917 - confidence_loss: 207.1518 - c49/98 [==============>...............] - ETA: 1:54 - loss: 207.9085 - location_loss: 2.3967 - confidence_loss: 203.3332 - c50/98 [==============>...............] - ETA: 1:51 - loss: 204.1457 - location_loss: 2.3994 - confidence_loss: 199.5783 - c51/98 [==============>...............] - ETA: 1:48 - loss: 200.4449 - location_loss: 2.3868 - confidence_loss: 195.9048 - c52/98 [==============>...............] - ETA: 1:45 - loss: 197.0547 - location_loss: 2.3998 - confidence_loss: 192.5049 - c53/98 [===============>..............] - ETA: 1:43 - loss: 193.5993 - location_loss: 2.3860 - confidence_loss: 189.0767 - c54/98 [===============>..............] - ETA: 1:40 - loss: 190.3777 - location_loss: 2.3853 - confidence_loss: 185.8620 - c55/98 [===============>..............] - ETA: 1:37 - loss: 187.2260 - location_loss: 2.3782 - confidence_loss: 182.7294 - c56/98 [================>.............] - ETA: 1:34 - loss: 184.2688 - location_loss: 2.3789 - confidence_loss: 179.7754 - c57/98 [================>.............] - ETA: 1:32 - loss: 181.3716 - location_loss: 2.3814 - confidence_loss: 176.8855 - c58/98 [================>.............] - ETA: 1:29 - loss: 178.6160 - location_loss: 2.3841 - confidence_loss: 174.1312 - c59/98 [=================>............] - ETA: 1:26 - loss: 175.8775 - location_loss: 2.3790 - confidence_loss: 171.4070 - c60/98 [=================>............] - ETA: 1:24 - loss: 173.2713 - location_loss: 2.3822 - confidence_loss: 168.8008 - c61/98 [=================>............] - ETA: 1:21 - loss: 170.6936 - location_loss: 2.3704 - confidence_loss: 166.2441 - c62/98 [=================>............] - ETA: 1:19 - loss: 168.2769 - location_loss: 2.3766 - confidence_loss: 163.8265 - c63/98 [==================>...........] - ETA: 1:16 - loss: 165.8532 - location_loss: 2.3670 - confidence_loss: 161.4231 - c64/98 [==================>...........] - ETA: 1:14 - loss: 163.5495 - location_loss: 2.3624 - confidence_loss: 159.1319 - c65/98 [==================>...........] - ETA: 1:11 - loss: 161.3247 - location_loss: 2.3595 - confidence_loss: 156.9159 - c66/98 [===================>..........] - ETA: 1:09 - loss: 159.1522 - location_loss: 2.3572 - confidence_loss: 154.7515 - c67/98 [===================>..........] - ETA: 1:06 - loss: 157.0995 - location_loss: 2.3629 - confidence_loss: 152.6961 - c68/98 [===================>..........] - ETA: 1:04 - loss: 155.0762 - location_loss: 2.3611 - confidence_loss: 150.6797 - c69/98 [====================>.........] - ETA: 1:02 - loss: 153.1389 - location_loss: 2.3665 - confidence_loss: 148.7404 - c70/98 [====================>.........] - ETA: 59s - loss: 151.2570 - location_loss: 2.3696 - confidence_loss: 146.8582 - cl71/98 [====================>.........] - ETA: 57s - loss: 149.3035 - location_loss: 2.3578 - confidence_loss: 144.9264 - cl72/98 [=====================>........] - ETA: 55s - loss: 147.5799 - location_loss: 2.3713 - confidence_loss: 143.1884 - cl73/98 [=====================>........] - ETA: 52s - loss: 145.8890 - location_loss: 2.3819 - confidence_loss: 141.4875 - cl74/98 [=====================>........] - ETA: 50s - loss: 144.1031 - location_loss: 2.3711 - confidence_loss: 139.7218 - cl75/98 [=====================>........] - ETA: 48s - loss: 142.4731 - location_loss: 2.3725 - confidence_loss: 138.0947 - cl76/98 [======================>.......] - ETA: 46s - loss: 140.8609 - location_loss: 2.3725 - confidence_loss: 136.4851 - cl77/98 [======================>.......] - ETA: 43s - loss: 139.2241 - location_loss: 2.3674 - confidence_loss: 134.8628 - cl78/98 [======================>.......] - ETA: 41s - loss: 137.6723 - location_loss: 2.3672 - confidence_loss: 133.3168 - cl79/98 [=======================>......] - ETA: 39s - loss: 136.1651 - location_loss: 2.3676 - confidence_loss: 131.8157 - cl80/98 [=======================>......] - ETA: 37s - loss: 134.6260 - location_loss: 2.3574 - confidence_loss: 130.2958 - cl81/98 [=======================>......] - ETA: 35s - loss: 133.2877 - location_loss: 2.3692 - confidence_loss: 128.9464 - cl82/98 [========================>.....] - ETA: 33s - loss: 131.8920 - location_loss: 2.3647 - confidence_loss: 127.5621 - cl83/98 [========================>.....] - ETA: 30s - loss: 130.4696 - location_loss: 2.3551 - confidence_loss: 126.1578 - cl84/98 [========================>.....] - ETA: 28s - loss: 129.1522 - location_loss: 2.3595 - confidence_loss: 124.8417 - cl85/98 [=========================>....] - ETA: 26s - loss: 127.8961 - location_loss: 2.3644 - confidence_loss: 123.5836 - cl86/98 [=========================>....] - ETA: 24s - loss: 126.5902 - location_loss: 2.3604 - confidence_loss: 122.2882 - cl87/98 [=========================>....] - ETA: 22s - loss: 125.3360 - location_loss: 2.3583 - confidence_loss: 121.0418 - cl88/98 [=========================>....] - ETA: 20s - loss: 124.0667 - location_loss: 2.3502 - confidence_loss: 119.7874 - cl89/98 [==========================>...] - ETA: 18s - loss: 122.8774 - location_loss: 2.3480 - confidence_loss: 118.6042 - cl90/98 [==========================>...] - ETA: 16s - loss: 121.6785 - location_loss: 2.3426 - confidence_loss: 117.4168 - cl91/98 [==========================>...] - ETA: 14s - loss: 120.4974 - location_loss: 2.3350 - confidence_loss: 116.2498 - cl92/98 [===========================>..] - ETA: 12s - loss: 119.4851 - location_loss: 2.3453 - confidence_loss: 115.2256 - cl93/98 [===========================>..] - ETA: 10s - loss: 118.4380 - location_loss: 2.3493 - confidence_loss: 114.1749 - cl94/98 [===========================>..] - ETA: 8s - loss: 117.4306 - location_loss: 2.3582 - confidence_loss: 113.1613 - cla95/98 [============================>.] - ETA: 6s - loss: 116.3741 - location_loss: 2.3541 - confidence_loss: 112.1134 - cla96/98 [============================>.] - ETA: 4s - loss: 115.3517 - location_loss: 2.3517 - confidence_loss: 111.0978 - cla97/98 [============================>.] - ETA: 1s - loss: 114.3009 - location_loss: 2.3449 - confidence_loss: 110.0607 - cla98/98 [==============================] - 216s 2s/step - loss: 113.3158 - location_loss: 2.3448 - confidence_loss: 109.0796- class_loss: 1.0238 - val_loss: 57.8813 - val_location_loss: 1.9132 - val_confidence_loss: 53.6828 - val_class_loss: 1.4178
Epoch 2/20
 1/98 [..............................] - ETA: 2:43 - loss: 14.4697 - location_loss: 1.8524 - confidence_loss: 11.4058 - cla 2/98 [..............................] - ETA: 2:40 - loss: 17.1683 - location_loss: 2.2128 - confidence_loss: 13.5890 - cla 3/98 [..............................] - ETA: 2:37 - loss: 19.8264 - location_loss: 2.5812 - confidence_loss: 15.7313 - cla 4/98 [>.............................] - ETA: 2:35 - loss: 18.5528 - location_loss: 2.3916 - confidence_loss: 14.6845 - cla 5/98 [>.............................] - ETA: 2:33 - loss: 17.4978 - location_loss: 2.2713 - confidence_loss: 13.7981 - cla 6/98 [>.............................] - ETA: 2:32 - loss: 17.6357 - location_loss: 2.3027 - confidence_loss: 13.9070 - cla 7/98 [=>............................] - ETA: 2:30 - loss: 17.8902 - location_loss: 2.3269 - confidence_loss: 14.1235 - cla 8/98 [=>............................] - ETA: 2:28 - loss: 17.3192 - location_loss: 2.2461 - confidence_loss: 13.6567 - cla 9/98 [=>............................] - ETA: 2:26 - loss: 17.1436 - location_loss: 2.2181 - confidence_loss: 13.5135 - cla10/98 [==>...........................] - ETA: 2:24 - loss: 17.4442 - location_loss: 2.2619 - confidence_loss: 13.7550 - cla11/98 [==>...........................] - ETA: 2:22 - loss: 17.4246 - location_loss: 2.2788 - confidence_loss: 13.7207 - cla12/98 [==>...........................] - ETA: 2:20 - loss: 17.6482 - location_loss: 2.3170 - confidence_loss: 13.8971 - cla13/98 [==>...........................] - ETA: 2:19 - loss: 17.9186 - location_loss: 2.3487 - confidence_loss: 14.1377 - cla14/98 [===>..........................] - ETA: 2:17 - loss: 17.4782 - location_loss: 2.2788 - confidence_loss: 13.7856 - cla15/98 [===>..........................] - ETA: 2:15 - loss: 17.2903 - location_loss: 2.2332 - confidence_loss: 13.6529 - cla16/98 [===>..........................] - ETA: 2:13 - loss: 17.5486 - location_loss: 2.2722 - confidence_loss: 13.8540 - cla17/98 [====>.........................] - ETA: 2:12 - loss: 17.5324 - location_loss: 2.2685 - confidence_loss: 13.8379 - cla18/98 [====>.........................] - ETA: 2:10 - loss: 17.4509 - location_loss: 2.2530 - confidence_loss: 13.7724 - cla19/98 [====>.........................] - ETA: 2:09 - loss: 17.6333 - location_loss: 2.2800 - confidence_loss: 13.9278 - cla20/98 [=====>........................] - ETA: 2:07 - loss: 17.7136 - location_loss: 2.2834 - confidence_loss: 13.9953 - cla21/98 [=====>........................] - ETA: 2:06 - loss: 17.4960 - location_loss: 2.2476 - confidence_loss: 13.8191 - cla22/98 [=====>........................] - ETA: 2:04 - loss: 17.4502 - location_loss: 2.2355 - confidence_loss: 13.7947 - cla23/98 [======>.......................] - ETA: 2:02 - loss: 17.2643 - location_loss: 2.2130 - confidence_loss: 13.6423 - cla24/98 [======>.......................] - ETA: 2:01 - loss: 17.1667 - location_loss: 2.1999 - confidence_loss: 13.5633 - cla25/98 [======>.......................] - ETA: 1:59 - loss: 17.2879 - location_loss: 2.2100 - confidence_loss: 13.6661 - cla26/98 [======>.......................] - ETA: 1:57 - loss: 17.1490 - location_loss: 2.1917 - confidence_loss: 13.5528 - cla27/98 [=======>......................] - ETA: 1:56 - loss: 17.0689 - location_loss: 2.1854 - confidence_loss: 13.4860 - cla28/98 [=======>......................] - ETA: 1:54 - loss: 17.1412 - location_loss: 2.2003 - confidence_loss: 13.5386 - cla29/98 [=======>......................] - ETA: 1:52 - loss: 17.1472 - location_loss: 2.2104 - confidence_loss: 13.5320 - cla30/98 [========>.....................] - ETA: 1:51 - loss: 17.0851 - location_loss: 2.2008 - confidence_loss: 13.4870 - cla31/98 [========>.....................] - ETA: 1:49 - loss: 17.0828 - location_loss: 2.2111 - confidence_loss: 13.4782 - cla32/98 [========>.....................] - ETA: 1:47 - loss: 17.2585 - location_loss: 2.2335 - confidence_loss: 13.6258 - cla33/98 [=========>....................] - ETA: 1:46 - loss: 17.4022 - location_loss: 2.2565 - confidence_loss: 13.7419 - cla34/98 [=========>....................] - ETA: 1:44 - loss: 17.3472 - location_loss: 2.2416 - confidence_loss: 13.7056 - cla35/98 [=========>....................] - ETA: 1:42 - loss: 17.5890 - location_loss: 2.2680 - confidence_loss: 13.9132 - cla36/98 [==========>...................] - ETA: 1:41 - loss: 17.5698 - location_loss: 2.2666 - confidence_loss: 13.8971 - cla37/98 [==========>...................] - ETA: 1:39 - loss: 17.4842 - location_loss: 2.2570 - confidence_loss: 13.8274 - cla38/98 [==========>...................] - ETA: 1:37 - loss: 17.4325 - location_loss: 2.2510 - confidence_loss: 13.7828 - cla39/98 [==========>...................] - ETA: 1:36 - loss: 17.3417 - location_loss: 2.2406 - confidence_loss: 13.7029 - cla40/98 [===========>..................] - ETA: 1:34 - loss: 17.3039 - location_loss: 2.2334 - confidence_loss: 13.6737 - cla41/98 [===========>..................] - ETA: 1:32 - loss: 17.1862 - location_loss: 2.2196 - confidence_loss: 13.5746 - cla42/98 [===========>..................] - ETA: 1:31 - loss: 17.1670 - location_loss: 2.2185 - confidence_loss: 13.5560 - cla43/98 [============>.................] - ETA: 1:29 - loss: 17.1848 - location_loss: 2.2237 - confidence_loss: 13.5655 - cla44/98 [============>.................] - ETA: 1:27 - loss: 17.2999 - location_loss: 2.2432 - confidence_loss: 13.6593 - cla45/98 [============>.................] - ETA: 1:26 - loss: 17.3369 - location_loss: 2.2508 - confidence_loss: 13.6886 - cla46/98 [=============>................] - ETA: 1:24 - loss: 17.2749 - location_loss: 2.2461 - confidence_loss: 13.6357 - cla47/98 [=============>................] - ETA: 1:23 - loss: 17.2623 - location_loss: 2.2499 - confidence_loss: 13.6210 - cla48/98 [=============>................] - ETA: 1:21 - loss: 17.2817 - location_loss: 2.2504 - confidence_loss: 13.6397 - cla49/98 [==============>...............] - ETA: 1:19 - loss: 17.2499 - location_loss: 2.2473 - confidence_loss: 13.6108 - cla50/98 [==============>...............] - ETA: 1:18 - loss: 17.3202 - location_loss: 2.2565 - confidence_loss: 13.6687 - cla51/98 [==============>...............] - ETA: 1:16 - loss: 17.4090 - location_loss: 2.2654 - confidence_loss: 13.7439 - cla52/98 [==============>...............] - ETA: 1:14 - loss: 17.4861 - location_loss: 2.2719 - confidence_loss: 13.8133 - cla53/98 [===============>..............] - ETA: 1:13 - loss: 17.4673 - location_loss: 2.2710 - confidence_loss: 13.7996 - cla54/98 [===============>..............] - ETA: 1:11 - loss: 17.5455 - location_loss: 2.2776 - confidence_loss: 13.8700 - cla55/98 [===============>..............] - ETA: 1:10 - loss: 17.5389 - location_loss: 2.2747 - confidence_loss: 13.8666 - cla56/98 [================>.............] - ETA: 1:08 - loss: 17.5063 - location_loss: 2.2719 - confidence_loss: 13.8405 - cla57/98 [================>.............] - ETA: 1:06 - loss: 17.3639 - location_loss: 2.2541 - confidence_loss: 13.7225 - cla58/98 [================>.............] - ETA: 1:05 - loss: 17.3467 - location_loss: 2.2547 - confidence_loss: 13.7064 - cla59/98 [=================>............] - ETA: 1:03 - loss: 17.4022 - location_loss: 2.2591 - confidence_loss: 13.7539 - cla60/98 [=================>............] - ETA: 1:01 - loss: 17.3830 - location_loss: 2.2590 - confidence_loss: 13.7368 - cla61/98 [=================>............] - ETA: 1:00 - loss: 17.2921 - location_loss: 2.2485 - confidence_loss: 13.6597 - cla62/98 [=================>............] - ETA: 58s - loss: 17.2233 - location_loss: 2.2426 - confidence_loss: 13.5990 - clas63/98 [==================>...........] - ETA: 57s - loss: 17.2716 - location_loss: 2.2488 - confidence_loss: 13.6395 - clas64/98 [==================>...........] - ETA: 55s - loss: 17.1967 - location_loss: 2.2376 - confidence_loss: 13.5800 - clas65/98 [==================>...........] - ETA: 53s - loss: 17.2257 - location_loss: 2.2425 - confidence_loss: 13.6030 - clas66/98 [===================>..........] - ETA: 52s - loss: 17.1766 - location_loss: 2.2395 - confidence_loss: 13.5606 - clas67/98 [===================>..........] - ETA: 50s - loss: 17.0869 - location_loss: 2.2272 - confidence_loss: 13.4881 - clas68/98 [===================>..........] - ETA: 48s - loss: 17.0434 - location_loss: 2.2187 - confidence_loss: 13.4554 - clas69/98 [====================>.........] - ETA: 47s - loss: 17.0184 - location_loss: 2.2172 - confidence_loss: 13.4348 - clas70/98 [====================>.........] - ETA: 45s - loss: 16.9790 - location_loss: 2.2155 - confidence_loss: 13.3989 - clas71/98 [====================>.........] - ETA: 44s - loss: 16.9904 - location_loss: 2.2201 - confidence_loss: 13.4040 - clas72/98 [=====================>........] - ETA: 42s - loss: 16.9821 - location_loss: 2.2213 - confidence_loss: 13.3955 - clas73/98 [=====================>........] - ETA: 40s - loss: 16.9507 - location_loss: 2.2199 - confidence_loss: 13.3670 - clas74/98 [=====================>........] - ETA: 39s - loss: 17.0255 - location_loss: 2.2290 - confidence_loss: 13.4288 - clas75/98 [=====================>........] - ETA: 37s - loss: 17.0071 - location_loss: 2.2222 - confidence_loss: 13.4196 - clas76/98 [======================>.......] - ETA: 35s - loss: 17.0542 - location_loss: 2.2266 - confidence_loss: 13.4614 - clas77/98 [======================>.......] - ETA: 34s - loss: 17.0644 - location_loss: 2.2339 - confidence_loss: 13.4649 - clas78/98 [======================>.......] - ETA: 32s - loss: 17.0805 - location_loss: 2.2394 - confidence_loss: 13.4755 - clas79/98 [=======================>......] - ETA: 30s - loss: 17.1195 - location_loss: 2.2473 - confidence_loss: 13.5059 - clas80/98 [=======================>......] - ETA: 29s - loss: 17.0774 - location_loss: 2.2445 - confidence_loss: 13.4685 - clas81/98 [=======================>......] - ETA: 27s - loss: 17.0379 - location_loss: 2.2422 - confidence_loss: 13.4347 - clas82/98 [========================>.....] - ETA: 26s - loss: 17.0143 - location_loss: 2.2414 - confidence_loss: 13.4132 - clas83/98 [========================>.....] - ETA: 24s - loss: 16.9936 - location_loss: 2.2410 - confidence_loss: 13.3940 - clas84/98 [========================>.....] - ETA: 22s - loss: 16.9904 - location_loss: 2.2411 - confidence_loss: 13.3923 - clas85/98 [=========================>....] - ETA: 21s - loss: 16.9788 - location_loss: 2.2412 - confidence_loss: 13.3824 - clas86/98 [=========================>....] - ETA: 19s - loss: 16.9352 - location_loss: 2.2352 - confidence_loss: 13.3465 - clas87/98 [=========================>....] - ETA: 17s - loss: 16.8731 - location_loss: 2.2293 - confidence_loss: 13.2930 - clas88/98 [=========================>....] - ETA: 16s - loss: 16.8466 - location_loss: 2.2239 - confidence_loss: 13.2730 - clas89/98 [==========================>...] - ETA: 14s - loss: 16.8717 - location_loss: 2.2257 - confidence_loss: 13.2948 - clas90/98 [==========================>...] - ETA: 13s - loss: 16.8759 - location_loss: 2.2296 - confidence_loss: 13.2980 - clas91/98 [==========================>...] - ETA: 11s - loss: 16.9128 - location_loss: 2.2323 - confidence_loss: 13.3319 - clas92/98 [===========================>..] - ETA: 9s - loss: 16.9024 - location_loss: 2.2297 - confidence_loss: 13.3249 - class93/98 [===========================>..] - ETA: 8s - loss: 16.8395 - location_loss: 2.2217 - confidence_loss: 13.2724 - class94/98 [===========================>..] - ETA: 6s - loss: 16.8239 - location_loss: 2.2188 - confidence_loss: 13.2608 - class95/98 [============================>.] - ETA: 4s - loss: 16.8550 - location_loss: 2.2238 - confidence_loss: 13.2869 - class96/98 [============================>.] - ETA: 3s - loss: 16.8999 - location_loss: 2.2300 - confidence_loss: 13.3251 - class97/98 [============================>.] - ETA: 1s - loss: 16.8464 - location_loss: 2.2190 - confidence_loss: 13.2858 - class98/98 [==============================] - 175s 2s/step - loss: 16.8092 - location_loss: 2.2135 - confidence_loss: 13.2564 -class_loss: 0.4718 - val_loss: 21.0042 - val_location_loss: 2.0677 - val_confidence_loss: 17.2026 - val_class_loss: 0.8663
Epoch 3/20
 1/98 [..............................] - ETA: 2:42 - loss: 12.8533 - location_loss: 1.6359 - confidence_loss: 10.0396 - cla 2/98 [..............................] - ETA: 2:38 - loss: 15.4273 - location_loss: 1.9679 - confidence_loss: 12.2023 - cla 3/98 [..............................] - ETA: 2:37 - loss: 15.7355 - location_loss: 2.0049 - confidence_loss: 12.4702 - cla 4/98 [>.............................] - ETA: 2:34 - loss: 16.7161 - location_loss: 2.1572 - confidence_loss: 13.2875 - cla 5/98 [>.............................] - ETA: 2:32 - loss: 18.5794 - location_loss: 2.4553 - confidence_loss: 14.7681 - cla 6/98 [>.............................] - ETA: 2:30 - loss: 18.0729 - location_loss: 2.3600 - confidence_loss: 14.3687 - cla 7/98 [=>............................] - ETA: 2:28 - loss: 17.9889 - location_loss: 2.3448 - confidence_loss: 14.3072 - cla 8/98 [=>............................] - ETA: 2:27 - loss: 18.0252 - location_loss: 2.3431 - confidence_loss: 14.3477 - cla 9/98 [=>............................] - ETA: 2:25 - loss: 17.6831 - location_loss: 2.3097 - confidence_loss: 14.0631 - cla10/98 [==>...........................] - ETA: 2:23 - loss: 17.5503 - location_loss: 2.2997 - confidence_loss: 13.9422 - cla11/98 [==>...........................] - ETA: 2:21 - loss: 16.8375 - location_loss: 2.2099 - confidence_loss: 13.3441 - cla12/98 [==>...........................] - ETA: 2:19 - loss: 16.4606 - location_loss: 2.1499 - confidence_loss: 13.0305 - cla13/98 [==>...........................] - ETA: 2:17 - loss: 16.5077 - location_loss: 2.1390 - confidence_loss: 13.0820 - cla14/98 [===>..........................] - ETA: 2:16 - loss: 16.4020 - location_loss: 2.1360 - confidence_loss: 12.9835 - cla15/98 [===>..........................] - ETA: 2:15 - loss: 16.5706 - location_loss: 2.1642 - confidence_loss: 13.1237 - cla16/98 [===>..........................] - ETA: 2:13 - loss: 16.4520 - location_loss: 2.1532 - confidence_loss: 13.0213 - cla17/98 [====>.........................] - ETA: 2:11 - loss: 16.0600 - location_loss: 2.1013 - confidence_loss: 12.6956 - cla18/98 [====>.........................] - ETA: 2:10 - loss: 15.9342 - location_loss: 2.0879 - confidence_loss: 12.5870 - cla19/98 [====>.........................] - ETA: 2:08 - loss: 15.9254 - location_loss: 2.0900 - confidence_loss: 12.5782 - cla20/98 [=====>........................] - ETA: 2:06 - loss: 16.0724 - location_loss: 2.1105 - confidence_loss: 12.7023 - cla21/98 [=====>........................] - ETA: 2:05 - loss: 16.0997 - location_loss: 2.1225 - confidence_loss: 12.7193 - cla22/98 [=====>........................] - ETA: 2:03 - loss: 16.3480 - location_loss: 2.1495 - confidence_loss: 12.9351 - cla23/98 [======>.......................] - ETA: 2:02 - loss: 16.4340 - location_loss: 2.1589 - confidence_loss: 13.0090 - cla24/98 [======>.......................] - ETA: 2:00 - loss: 16.2265 - location_loss: 2.1256 - confidence_loss: 12.8379 - cla25/98 [======>.......................] - ETA: 1:59 - loss: 16.2156 - location_loss: 2.1236 - confidence_loss: 12.8263 - cla26/98 [======>.......................] - ETA: 1:57 - loss: 16.2020 - location_loss: 2.1327 - confidence_loss: 12.8096 - cla27/98 [=======>......................] - ETA: 1:55 - loss: 16.2389 - location_loss: 2.1430 - confidence_loss: 12.8325 - cla28/98 [=======>......................] - ETA: 1:54 - loss: 16.4442 - location_loss: 2.1642 - confidence_loss: 13.0123 - cla29/98 [=======>......................] - ETA: 1:52 - loss: 16.4493 - location_loss: 2.1711 - confidence_loss: 13.0110 - cla30/98 [========>.....................] - ETA: 1:50 - loss: 16.3366 - location_loss: 2.1579 - confidence_loss: 12.9160 - cla31/98 [========>.....................] - ETA: 1:49 - loss: 16.3224 - location_loss: 2.1513 - confidence_loss: 12.9084 - cla32/98 [========>.....................] - ETA: 1:47 - loss: 16.2989 - location_loss: 2.1473 - confidence_loss: 12.8905 - cla33/98 [=========>....................] - ETA: 1:45 - loss: 16.3098 - location_loss: 2.1548 - confidence_loss: 12.8945 - cla34/98 [=========>....................] - ETA: 1:44 - loss: 16.2719 - location_loss: 2.1503 - confidence_loss: 12.8625 - cla35/98 [=========>....................] - ETA: 1:42 - loss: 16.2781 - location_loss: 2.1554 - confidence_loss: 12.8628 - cla36/98 [==========>...................] - ETA: 1:40 - loss: 16.2991 - location_loss: 2.1604 - confidence_loss: 12.8814 - cla37/98 [==========>...................] - ETA: 1:39 - loss: 16.2985 - location_loss: 2.1601 - confidence_loss: 12.8830 - cla38/98 [==========>...................] - ETA: 1:37 - loss: 16.3683 - location_loss: 2.1668 - confidence_loss: 12.9444 - cla39/98 [==========>...................] - ETA: 1:36 - loss: 16.3473 - location_loss: 2.1704 - confidence_loss: 12.9218 - cla40/98 [===========>..................] - ETA: 1:34 - loss: 16.4498 - location_loss: 2.1846 - confidence_loss: 13.0065 - cla41/98 [===========>..................] - ETA: 1:32 - loss: 16.5541 - location_loss: 2.1992 - confidence_loss: 13.0960 - cla42/98 [===========>..................] - ETA: 1:31 - loss: 16.4891 - location_loss: 2.1945 - confidence_loss: 13.0392 - cla43/98 [============>.................] - ETA: 1:29 - loss: 16.4558 - location_loss: 2.1932 - confidence_loss: 13.0099 - cla44/98 [============>.................] - ETA: 1:27 - loss: 16.3130 - location_loss: 2.1702 - confidence_loss: 12.8950 - cla45/98 [============>.................] - ETA: 1:26 - loss: 16.2259 - location_loss: 2.1590 - confidence_loss: 12.8229 - cla46/98 [=============>................] - ETA: 1:24 - loss: 16.1676 - location_loss: 2.1528 - confidence_loss: 12.7726 - cla47/98 [=============>................] - ETA: 1:22 - loss: 16.2010 - location_loss: 2.1600 - confidence_loss: 12.7974 - cla48/98 [=============>................] - ETA: 1:21 - loss: 16.2614 - location_loss: 2.1665 - confidence_loss: 12.8469 - cla49/98 [==============>...............] - ETA: 1:19 - loss: 16.2086 - location_loss: 2.1592 - confidence_loss: 12.8029 - cla50/98 [==============>...............] - ETA: 1:18 - loss: 16.2379 - location_loss: 2.1640 - confidence_loss: 12.8262 - cla51/98 [==============>...............] - ETA: 1:16 - loss: 16.2285 - location_loss: 2.1630 - confidence_loss: 12.8195 - cla52/98 [==============>...............] - ETA: 1:14 - loss: 16.1543 - location_loss: 2.1542 - confidence_loss: 12.7573 - cla53/98 [===============>..............] - ETA: 1:13 - loss: 16.0752 - location_loss: 2.1436 - confidence_loss: 12.6929 - cla54/98 [===============>..............] - ETA: 1:11 - loss: 16.1087 - location_loss: 2.1459 - confidence_loss: 12.7224 - cla55/98 [===============>..............] - ETA: 1:09 - loss: 16.0984 - location_loss: 2.1437 - confidence_loss: 12.7137 - cla56/98 [================>.............] - ETA: 1:08 - loss: 16.0514 - location_loss: 2.1356 - confidence_loss: 12.6773 - cla57/98 [================>.............] - ETA: 1:06 - loss: 15.9824 - location_loss: 2.1285 - confidence_loss: 12.6179 - cla58/98 [================>.............] - ETA: 1:05 - loss: 15.9375 - location_loss: 2.1261 - confidence_loss: 12.5778 - cla59/98 [=================>............] - ETA: 1:03 - loss: 15.9168 - location_loss: 2.1259 - confidence_loss: 12.5581 - cla60/98 [=================>............] - ETA: 1:01 - loss: 15.8968 - location_loss: 2.1230 - confidence_loss: 12.5413 - cla61/98 [=================>............] - ETA: 1:00 - loss: 15.9020 - location_loss: 2.1237 - confidence_loss: 12.5468 - cla62/98 [=================>............] - ETA: 58s - loss: 15.8856 - location_loss: 2.1214 - confidence_loss: 12.5326 - clas63/98 [==================>...........] - ETA: 56s - loss: 15.8653 - location_loss: 2.1193 - confidence_loss: 12.5156 - clas64/98 [==================>...........] - ETA: 55s - loss: 15.8888 - location_loss: 2.1242 - confidence_loss: 12.5343 - clas65/98 [==================>...........] - ETA: 53s - loss: 15.9440 - location_loss: 2.1363 - confidence_loss: 12.5768 - clas66/98 [===================>..........] - ETA: 52s - loss: 15.9224 - location_loss: 2.1324 - confidence_loss: 12.5597 - clas67/98 [===================>..........] - ETA: 50s - loss: 15.8846 - location_loss: 2.1273 - confidence_loss: 12.5281 - clas68/98 [===================>..........] - ETA: 48s - loss: 15.8811 - location_loss: 2.1270 - confidence_loss: 12.5252 - clas69/98 [====================>.........] - ETA: 47s - loss: 15.8207 - location_loss: 2.1166 - confidence_loss: 12.4776 - clas70/98 [====================>.........] - ETA: 45s - loss: 15.7583 - location_loss: 2.1093 - confidence_loss: 12.4254 - clas71/98 [====================>.........] - ETA: 43s - loss: 15.7558 - location_loss: 2.1098 - confidence_loss: 12.4228 - clas72/98 [=====================>........] - ETA: 42s - loss: 15.7436 - location_loss: 2.1051 - confidence_loss: 12.4161 - clas73/98 [=====================>........] - ETA: 40s - loss: 15.7326 - location_loss: 2.1072 - confidence_loss: 12.4040 - clas74/98 [=====================>........] - ETA: 39s - loss: 15.7507 - location_loss: 2.1103 - confidence_loss: 12.4203 - clas75/98 [=====================>........] - ETA: 37s - loss: 15.7095 - location_loss: 2.1054 - confidence_loss: 12.3852 - clas76/98 [======================>.......] - ETA: 35s - loss: 15.6945 - location_loss: 2.1018 - confidence_loss: 12.3756 - clas77/98 [======================>.......] - ETA: 34s - loss: 15.6234 - location_loss: 2.0903 - confidence_loss: 12.3176 - clas78/98 [======================>.......] - ETA: 32s - loss: 15.6298 - location_loss: 2.0905 - confidence_loss: 12.3234 - clas79/98 [=======================>......] - ETA: 30s - loss: 15.6092 - location_loss: 2.0862 - confidence_loss: 12.3082 - clas80/98 [=======================>......] - ETA: 29s - loss: 15.6136 - location_loss: 2.0851 - confidence_loss: 12.3138 - clas81/98 [=======================>......] - ETA: 27s - loss: 15.5900 - location_loss: 2.0798 - confidence_loss: 12.2963 - clas82/98 [========================>.....] - ETA: 25s - loss: 15.6557 - location_loss: 2.0889 - confidence_loss: 12.3522 - clas83/98 [========================>.....] - ETA: 24s - loss: 15.6399 - location_loss: 2.0876 - confidence_loss: 12.3386 - clas84/98 [========================>.....] - ETA: 22s - loss: 15.6201 - location_loss: 2.0857 - confidence_loss: 12.3222 - clas85/98 [=========================>....] - ETA: 21s - loss: 15.5950 - location_loss: 2.0835 - confidence_loss: 12.3015 - clas86/98 [=========================>....] - ETA: 19s - loss: 15.5519 - location_loss: 2.0791 - confidence_loss: 12.2653 - clas87/98 [=========================>....] - ETA: 17s - loss: 15.5497 - location_loss: 2.0780 - confidence_loss: 12.2645 - clas88/98 [=========================>....] - ETA: 16s - loss: 15.5711 - location_loss: 2.0807 - confidence_loss: 12.2830 - clas89/98 [==========================>...] - ETA: 14s - loss: 15.5458 - location_loss: 2.0754 - confidence_loss: 12.2648 - clas90/98 [==========================>...] - ETA: 13s - loss: 15.5495 - location_loss: 2.0767 - confidence_loss: 12.2678 - clas91/98 [==========================>...] - ETA: 11s - loss: 15.5270 - location_loss: 2.0745 - confidence_loss: 12.2488 - clas92/98 [===========================>..] - ETA: 9s - loss: 15.5662 - location_loss: 2.0821 - confidence_loss: 12.2800 - class93/98 [===========================>..] - ETA: 8s - loss: 15.5817 - location_loss: 2.0859 - confidence_loss: 12.2910 - class94/98 [===========================>..] - ETA: 6s - loss: 15.5955 - location_loss: 2.0901 - confidence_loss: 12.3011 - class95/98 [============================>.] - ETA: 4s - loss: 15.5706 - location_loss: 2.0902 - confidence_loss: 12.2778 - class96/98 [============================>.] - ETA: 3s - loss: 15.5467 - location_loss: 2.0896 - confidence_loss: 12.2557 - class97/98 [============================>.] - ETA: 1s - loss: 15.5415 - location_loss: 2.0903 - confidence_loss: 12.2503 - class_loss: 0.3333v5_decode: False, elim_grid_sense: False
v5_decode: False, elim_grid_sense: False
Eval model: 100%|████████████████████████████████████████████████████████████████████████| 350/350 [00:41<00:00,  8.41it/s]

Pascal VOC AP evaluation
motorbike: AP 0.1302, precision 0.8210, recall 0.1494
mAP@IoU=0.40 result: 13.021417
mPrec@IoU=0.40 result: 82.098765
mRec@IoU=0.40 result: 14.943820

tf2.4

root@3c1ae4326e01:/workspace/tony/keras-YOLOv3-model-set# python train.py 
--model_type=yolo4_darknet 
--anchors_path=anchors/motorbike_yolov4_anchor_csp_onlycoco_416.txt 
--annotation_file=object_dataset/combine_annotation.txt 
--classes_path=classes/motorbike_classes.txt 
--gpu_num=1 
--batch_size=32  
--save_eval_checkpoint 
--optimizer=sgd 
--freeze_level=1 
--model_image_size=416x416 
--eval_online 
--eval_epoch_interval=3

2021-02-23 01:35:08.824440: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-23 01:35:11.896511: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-23 01:35:11.897772: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-02-23 01:35:16.371465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:61:00.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2021-02-23 01:35:16.371514: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-23 01:35:16.376544: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-23 01:35:16.376642: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-02-23 01:35:16.378299: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-23 01:35:16.378685: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-23 01:35:16.383042: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-23 01:35:16.384036: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-02-23 01:35:16.384230: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-23 01:35:16.386831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-23 01:35:16.819371: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2021-02-23 01:35:16.819419: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2021-02-23 01:35:16.819476: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs
2021-02-23 01:35:16.820680: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcupti.so.11.0
2021-02-23 01:35:16.844414: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2021-02-23 01:35:16.844653: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1487] CUPTI activity buffer flushed
using default seed: 42
2021-02-23 01:35:16.997003: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-02-23 01:35:17.049871: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-23 01:35:17.051480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:61:00.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2021-02-23 01:35:17.051522: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-23 01:35:17.051582: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-23 01:35:17.051602: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-02-23 01:35:17.051620: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-23 01:35:17.051639: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-23 01:35:17.051657: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-23 01:35:17.051676: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-02-23 01:35:17.051694: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-23 01:35:17.054206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-23 01:35:17.054252: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-23 01:35:17.879325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-23 01:35:17.879375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2021-02-23 01:35:17.879393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2021-02-23 01:35:17.883417: E tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] The TF_FORCE_GPU_ALLOW_GROWTH environment variable is set but could not be parsed: "True". Valid values are "true" or "false". Using original config value of 1.
2021-02-23 01:35:17.883470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14759 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:61:00.0, compute capability: 7.0)
backbone layers number: 250
Load weights weights/yolov4.h5.
Create  yolo4_darknet model with 9 anchors and 1 classes.
model layer number: 370
Freeze the first 250 layers of total 370 layers.
Transfer training stage
Train on 3152 samples, val on 350 samples, with batch size 32, input_shape (416, 416).
2021-02-23 01:35:23.743291: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-02-23 01:35:23.749476: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2000000000 Hz
Epoch 1/20
2021-02-23 01:35:53.055383: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-23 01:35:55.560918: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-23 01:35:56.028273: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
 1/98 [..............................] - ETA: 58:23 - loss: 7577.8179 - location_loss: 2.3624 - confidence_loss: 7572.9077 - class_loss: 1.68242021-02-23 01:35:59.889519: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2021-02-23 01:35:59.889559: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
 2/98 [..............................] - ETA: 4:11 - loss: 5692.6398 - location_loss: 2.5993 - confidence_loss: 3802.1077 - class_loss: 1.8886 2021-02-23 01:36:02.511893: I tensorflow/core/profiler/lib/profiler_session.cc:71] Profiler session collecting data.
2021-02-23 01:36:02.528737: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1487] CUPTI activity buffer flushed
2021-02-23 01:36:02.785889: I tensorflow/core/profiler/internal/gpu/cupti_collector.cc:228]  GpuTracer has collected 15688 callback api events and 14988 activity events.
2021-02-23 01:36:03.270821: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
 3/98 [..............................] - ETA: 4:14 - loss: 4644.9573 - location_loss: 2.6893 - confidence_loss: 2544.2146 - class_loss: 1.8 4/98 [>.............................] - ETA: 3:27 - loss: 3963.3198 - location_loss: 2.4441 - confidence_loss: 1913.3931 - class_loss: 1.7 5/98 [>.............................] - ETA: 3:04 - loss: 3479.6595 - location_loss: 2.4829 - confidence_loss: 1539.8839 - class_loss: 1.7 6/98 [>.............................] - ETA: 2:48 - loss: 3116.0034 - location_loss: 2.6541 - confidence_loss: 1292.2859 - class_loss: 1.9 7/98 [=>............................] - ETA: 2:37 - loss: 2830.2993 - location_loss: 2.5994 - confidence_loss: 1110.7391 - class_loss: 1.8 8/98 [=>............................] - ETA: 2:30 - loss: 2598.9155 - location_loss: 2.5231 - confidence_loss: 974.0292 - class_loss: 1.80 9/98 [=>............................] - ETA: 2:23 - loss: 2407.6940 - location_loss: 2.5714 - confidence_loss: 872.6699 - class_loss: 1.8110/98 [==>...........................] - ETA: 2:18 - loss: 2246.2399 - location_loss: 2.5095 - confidence_loss: 788.0018 - class_loss: 1.7711/98 [==>...........................] - ETA: 2:14 - loss: 2107.7986 - location_loss: 2.4954 - confidence_loss: 718.2695 - class_loss: 1.7512/98 [==>...........................] - ETA: 2:10 - loss: 1987.7381 - location_loss: 2.5014 - confidence_loss: 661.9735 - class_loss: 1.7313/98 [==>...........................] - ETA: 2:07 - loss: 1882.3810 - location_loss: 2.5350 - confidence_loss: 612.9603 - class_loss: 1.7314/98 [===>..........................] - ETA: 2:04 - loss: 1789.0650 - location_loss: 2.5535 - confidence_loss: 570.8129 - class_loss: 1.7215/98 [===>..........................] - ETA: 2:01 - loss: 1705.8017 - location_loss: 2.6046 - confidence_loss: 534.9094 - class_loss: 1.7316/98 [===>..........................] - ETA: 1:58 - loss: 1630.9383 - location_loss: 2.5714 - confidence_loss: 502.8340 - class_loss: 1.7117/98 [====>.........................] - ETA: 1:56 - loss: 1563.2248 - location_loss: 2.5965 - confidence_loss: 474.6321 - class_loss: 1.7118/98 [====>.........................] - ETA: 1:53 - loss: 1501.6314 - location_loss: 2.5775 - confidence_loss: 449.3930 - class_loss: 1.7019/98 [====>.........................] - ETA: 1:51 - loss: 1445.3287 - location_loss: 2.5547 - confidence_loss: 426.7754 - class_loss: 1.6820/98 [=====>........................] - ETA: 1:49 - loss: 1393.6446 - location_loss: 2.5488 - confidence_loss: 406.5510 - class_loss: 1.6721/98 [=====>........................] - ETA: 1:47 - loss: 1346.0092 - location_loss: 2.5462 - confidence_loss: 388.2245 - class_loss: 1.6622/98 [=====>........................] - ETA: 1:45 - loss: 1301.9397 - location_loss: 2.5358 - confidence_loss: 371.4264 - class_loss: 1.6523/98 [======>.......................] - ETA: 1:43 - loss: 1261.1942 - location_loss: 2.5023 - confidence_loss: 359.8029 - class_loss: 1.6224/98 [======>.......................] - ETA: 1:41 - loss: 1223.2581 - location_loss: 2.5252 - confidence_loss: 345.7150 - class_loss: 1.6125/98 [======>.......................] - ETA: 1:39 - loss: 1187.8279 - location_loss: 2.4891 - confidence_loss: 332.5530 - class_loss: 1.5926/98 [======>.......................] - ETA: 1:38 - loss: 1154.6686 - location_loss: 2.4853 - confidence_loss: 320.7518 - class_loss: 1.5827/98 [=======>......................] - ETA: 1:36 - loss: 1123.5835 - location_loss: 2.5226 - confidence_loss: 310.3785 - class_loss: 1.6028/98 [=======>......................] - ETA: 1:34 - loss: 1094.3422 - location_loss: 2.4984 - confidence_loss: 299.8794 - class_loss: 1.5829/98 [=======>......................] - ETA: 1:33 - loss: 1066.8009 - location_loss: 2.4902 - confidence_loss: 290.7042 - class_loss: 1.5830/98 [========>.....................] - ETA: 1:31 - loss: 1040.7988 - location_loss: 2.5028 - confidence_loss: 281.7857 - class_loss: 1.5831/98 [========>.....................] - ETA: 1:30 - loss: 1016.1948 - location_loss: 2.4694 - confidence_loss: 273.1860 - class_loss: 1.5532/98 [========>.....................] - ETA: 1:28 - loss: 992.8776 - location_loss: 2.4559 - confidence_loss: 265.1807 - class_loss: 1.53933/98 [=========>....................] - ETA: 1:27 - loss: 970.7580 - location_loss: 2.4781 - confidence_loss: 258.0385 - class_loss: 1.54834/98 [=========>....................] - ETA: 1:25 - loss: 949.7279 - location_loss: 2.4547 - confidence_loss: 250.8850 - class_loss: 1.52835/98 [=========>....................] - ETA: 1:24 - loss: 929.7052 - location_loss: 2.4356 - confidence_loss: 244.1206 - class_loss: 1.51036/98 [==========>...................] - ETA: 1:22 - loss: 910.6218 - location_loss: 2.4328 - confidence_loss: 237.9029 - class_loss: 1.50237/98 [==========>...................] - ETA: 1:21 - loss: 892.4060 - location_loss: 2.4150 - confidence_loss: 231.8681 - class_loss: 1.48438/98 [==========>...................] - ETA: 1:19 - loss: 875.0012 - location_loss: 2.4029 - confidence_loss: 226.2878 - class_loss: 1.46839/98 [==========>...................] - ETA: 1:18 - loss: 858.3522 - location_loss: 2.3831 - confidence_loss: 220.9803 - class_loss: 1.45740/98 [===========>..................] - ETA: 1:16 - loss: 842.4131 - location_loss: 2.3984 - confidence_loss: 216.0657 - class_loss: 1.45941/98 [===========>..................] - ETA: 1:15 - loss: 827.1366 - location_loss: 2.4019 - confidence_loss: 211.3484 - class_loss: 1.45742/98 [===========>..................] - ETA: 1:13 - loss: 812.4740 - location_loss: 2.3859 - confidence_loss: 206.6191 - class_loss: 1.43743/98 [============>.................] - ETA: 1:12 - loss: 798.3908 - location_loss: 2.3827 - confidence_loss: 202.2237 - class_loss: 1.42444/98 [============>.................] - ETA: 1:11 - loss: 784.8573 - location_loss: 2.3887 - confidence_loss: 198.2366 - class_loss: 1.42145/98 [============>.................] - ETA: 1:09 - loss: 771.8400 - location_loss: 2.3896 - confidence_loss: 194.4172 - class_loss: 1.40846/98 [=============>................] - ETA: 1:08 - loss: 759.3088 - location_loss: 2.3896 - confidence_loss: 190.7445 - class_loss: 1.40347/98 [=============>................] - ETA: 1:07 - loss: 747.2363 - location_loss: 2.3971 - confidence_loss: 187.2372 - class_loss: 1.40148/98 [=============>................] - ETA: 1:05 - loss: 735.6123 - location_loss: 2.4022 - confidence_loss: 184.6028 - class_loss: 1.41149/98 [==============>...............] - ETA: 1:04 - loss: 724.3935 - location_loss: 2.4006 - confidence_loss: 181.2227 - class_loss: 1.40150/98 [==============>...............] - ETA: 1:02 - loss: 713.5576 - location_loss: 2.3953 - confidence_loss: 177.9438 - class_loss: 1.39151/98 [==============>...............] - ETA: 1:01 - loss: 703.0834 - location_loss: 2.3863 - confidence_loss: 174.7420 - class_loss: 1.38052/98 [==============>...............] - ETA: 1:00 - loss: 692.9569 - location_loss: 2.4012 - confidence_loss: 171.8533 - class_loss: 1.38153/98 [===============>..............] - ETA: 58s - loss: 683.1550 - location_loss: 2.3827 - confidence_loss: 168.8423 - class_loss: 1.367498/98 [==============================] - 181s 1s/step - loss: 426.6083 - location_loss: 2.3245 - confidence_loss: 100.3958 - class_loss: 1.1663 - val_loss: 73.8888 - val_location_loss: 2.1186 - val_confidence_loss: 69.6127 - val_class_loss: 1.2907
Epoch 2/20
98/98 [==============================] - 136s 1s/step - loss: 20.4257 - location_loss: 2.0872 - confidence_loss: 17.4091 - class_loss: 0.6536 - val_loss: 23.8483 - val_location_loss: 1.9106 - val_confidence_loss: 20.2279 - val_class_loss: 0.8430
Epoch 3/20
98/98 [==============================] - 135s 1s/step - loss: 20.1851 - location_loss: 1.9848 - confidence_loss: 17.1292 - class_loss: 0.5107 - val_loss: 20.3556 - val_location_loss: 1.8659 - val_confidence_loss: 17.1231 - val_class_loss: 0.4997
v5_decode: False, elim_grid_sense: False
Eval model: 100%|████████████████████████████████████████████████████████████████████████████████████████| 350/350 [00:32<00:00, 10.74it/s]

Pascal VOC AP evaluation
motorbike: AP 0.0000, precision 0.0000, recall 0.0000
mAP@IoU=0.40 result: 0.000000
mPrec@IoU=0.40 result: 0.000000
mRec@IoU=0.40 result: 0.000000
david8862 commented 3 years ago

@tangyugu seems you're training "yolo4_darknet" model, which doesn't have Imagenet pretrained backbone weights in this repo and may lead to bad performance. You can try some other backbone type which have pretrained backbone like "yolo4_efficientnet" or "yolo4_mobilenetv2"