search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.74k stars 7.96k forks source link

Training loss fluctuates and the mAP0.5 is stuck at 0.35% #7053

Open ichigo97 opened 3 years ago

ichigo97 commented 3 years ago

@AlexeyAB Hi, I am trying to perform object detection on VisDrone Dataset 2020. I have 12 classes and set my max_batches to 8000(I Know that's pretty low but owing to time contsraints and others) and filters in conv. layers to 51. I am using Google colab GPU. My issue is that the Average loss fluctuates between 300 to 500, say it is 312 for an iteration then it goes up like 350, then 380 and upwards suddenly reducing again to 340 and 310 and then shoots up and my mAP@0.5 is always at 0.35% there is no observed increase or decrease. So I stopped it mid-way, so can you suggest any way to reduce the average loss and increase mAP@0.5. I'm thinking of adjusting the anchor box dimensions but I'm not sure how to go about it. Any other ideas are also welcome. I am attaching some screenshots to give some more context. image

(next mAP calculation at 3234 iterations) Last accuracy mAP@0.5 = 0.35 %, best = 0.35 % 2647: 518.109131, 470.850464 avg loss, 0.001000 rate, 2.496970 seconds, 42352 images, 3.733943 hours left Loaded: 0.000023 seconds v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.574499), count: 231, total_loss = 3225.596924 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.628248), count: 7, total_loss = 21.934875 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.002022 total_bbox = 4337519, rewritten_bbox = 7.680335 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.599994), count: 98, total_loss = 1048.430664 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.795191), count: 48, total_loss = 112.473076 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.504591), count: 2, total_loss = 1.644850 total_bbox = 4337667, rewritten_bbox = 7.680142 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.533508), count: 127, total_loss = 1969.805908 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.686571), count: 8, total_loss = 15.117030 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000084 total_bbox = 4337802, rewritten_bbox = 7.680180 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.448053), count: 43, total_loss = 399.957581 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.651049), count: 13, total_loss = 37.891514 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.505495), count: 6, total_loss = 12.893992 total_bbox = 4337864, rewritten_bbox = 7.680140 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.595773), count: 53, total_loss = 1028.500000 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.760069), count: 6, total_loss = 9.707025 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000198 total_bbox = 4337923, rewritten_bbox = 7.680127 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.545385), count: 181, total_loss = 2964.210938 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.686368), count: 22, total_loss = 51.143288 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.269464), count: 1, total_loss = 0.156002 total_bbox = 4338127, rewritten_bbox = 7.680227 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.667288), count: 53, total_loss = 446.475006 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.743717), count: 23, total_loss = 66.312065 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.013334 total_bbox = 4338203, rewritten_bbox = 7.680161 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.637692), count: 46, total_loss = 904.970703 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 1, total_loss = 0.084397 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000145 total_bbox = 4338249, rewritten_bbox = 7.680149 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.514178), count: 169, total_loss = 2199.572266 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.642248), count: 6, total_loss = 13.438636 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000275 total_bbox = 4338424, rewritten_bbox = 7.680254 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.576778), count: 188, total_loss = 2485.805908 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.682046), count: 22, total_loss = 57.968258 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.083724 total_bbox = 4338634, rewritten_bbox = 7.680413 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.570679), count: 68, total_loss = 946.913025 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.550614), count: 5, total_loss = 14.532759 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.059939 total_bbox = 4338707, rewritten_bbox = 7.680330 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.681728), count: 2, total_loss = 56.766277 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 1, total_loss = 0.022464 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.008291 total_bbox = 4338709, rewritten_bbox = 7.680326 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.613175), count: 94, total_loss = 1331.535034 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.673079), count: 11, total_loss = 26.789101 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.466967), count: 3, total_loss = 4.480790 total_bbox = 4338817, rewritten_bbox = 7.680388 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.503135), count: 45, total_loss = 640.022339 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 1, total_loss = 0.004207 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000025 total_bbox = 4338862, rewritten_bbox = 7.680516 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.593150), count: 253, total_loss = 4313.919434 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.717992), count: 9, total_loss = 28.266832 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000814 total_bbox = 4339124, rewritten_bbox = 7.680928 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.680873), count: 82, total_loss = 1603.556641 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.773061), count: 6, total_loss = 20.043634 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000088 total_bbox = 4339212, rewritten_bbox = 7.680841 % (next mAP calculation at 3234 iterations) Last accuracy mAP@0.5 = 0.35 %, best = 0.35 % 2648: 542.939819, 478.059387 avg loss, 0.001000 rate, 2.530839 seconds, 42368 images, 3.733733 hours left Loaded: 0.000026 seconds v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.689380), count: 60, total_loss = 697.846924 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.727533), count: 30, total_loss = 60.551762 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.348490), count: 2, total_loss = 3.251886 total_bbox = 4339304, rewritten_bbox = 7.680702 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.695966), count: 77, total_loss = 829.845947 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.724076), count: 23, total_loss = 52.026955 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.311896), count: 1, total_loss = 0.068989 total_bbox = 4339405, rewritten_bbox = 7.680523 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.658242), count: 31, total_loss = 366.773468 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.615570), count: 9, total_loss = 23.067677 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.333221), count: 1, total_loss = 2.650399 total_bbox = 4339446, rewritten_bbox = 7.680473 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.422563), count: 24, total_loss = 373.157532 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 1, total_loss = 0.001587 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000030 total_bbox = 4339470, rewritten_bbox = 7.680500 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.605483), count: 39, total_loss = 569.330200 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.739182), count: 1, total_loss = 3.531628 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000021 total_bbox = 4339510, rewritten_bbox = 7.680521 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.515463), count: 64, total_loss = 1224.949219 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 1, total_loss = 0.188118 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000050 total_bbox = 4339574, rewritten_bbox = 7.680523 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.752813), count: 60, total_loss = 579.071655 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.787360), count: 44, total_loss = 107.082771 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 1.374165 total_bbox = 4339678, rewritten_bbox = 7.680339 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.540643), count: 21, total_loss = 270.878387 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.666791), count: 5, total_loss = 12.744651 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.001243 total_bbox = 4339704, rewritten_bbox = 7.680316 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.433088), count: 207, total_loss = 2075.023193 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.769933), count: 8, total_loss = 28.305147 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000073 total_bbox = 4339919, rewritten_bbox = 7.681088 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.653640), count: 47, total_loss = 393.992004 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.713180), count: 31, total_loss = 65.692764 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.689720), count: 1, total_loss = 2.065495 total_bbox = 4339998, rewritten_bbox = 7.680948 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.603985), count: 61, total_loss = 444.019073 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.737008), count: 34, total_loss = 83.693176 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.533600), count: 4, total_loss = 5.923620 total_bbox = 4340097, rewritten_bbox = 7.680819 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.537365), count: 162, total_loss = 2058.548584 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.704038), count: 25, total_loss = 82.523483 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.029469 total_bbox = 4340284, rewritten_bbox = 7.681041 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.683720), count: 43, total_loss = 712.811951 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.513299), count: 4, total_loss = 21.458683 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.457467), count: 2, total_loss = 2.575466 total_bbox = 4340333, rewritten_bbox = 7.680954 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.681391), count: 78, total_loss = 1293.064453 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.786696), count: 3, total_loss = 15.752063 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000074 total_bbox = 4340414, rewritten_bbox = 7.680903 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.694670), count: 83, total_loss = 1163.762817 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.711150), count: 22, total_loss = 46.540367 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.474770), count: 7, total_loss = 7.768686 total_bbox = 4340526, rewritten_bbox = 7.680705 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.445187), count: 103, total_loss = 1394.976562 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.806088), count: 3, total_loss = 21.078218 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000049 total_bbox = 4340632, rewritten_bbox = 7.680840 % (next mAP calculation at 3234 iterations) Last accuracy mAP@0.5 = 0.35 %, best = 0.35 % 2649: 314.541626, 461.707611 avg loss, 0.001000 rate, 1.898425 seconds, 42384 images, 3.734023 hours left Loaded: 0.000050 seconds v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.347685), count: 69, total_loss = 1224.725220 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 1, total_loss = 0.025087 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.001477 total_bbox = 4340701, rewritten_bbox = 7.681064 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.574957), count: 78, total_loss = 1226.875854 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.552122), count: 5, total_loss = 13.617881 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000349 total_bbox = 4340784, rewritten_bbox = 7.681078 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.627577), count: 25, total_loss = 447.852142 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 1, total_loss = 0.004678 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000031 total_bbox = 4340809, rewritten_bbox = 7.681056 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.679054), count: 33, total_loss = 412.043732 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.000000), count: 1, total_loss = 0.004365 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000041 total_bbox = 4340842, rewritten_bbox = 7.680998 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.608791), count: 38, total_loss = 534.629761 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.634998), count: 4, total_loss = 22.075537 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.000000), count: 1, total_loss = 0.000080 total_bbox = 4340884, rewritten_bbox = 7.680947 % v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 139 Avg (IOU: 0.719992), count: 25, total_loss = 273.544952 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 150 Avg (IOU: 0.735634), count: 9, total_loss = 21.461012 v3 (iou loss, Normalizer: (iou: 0.07, obj: 1.00, cls: 1.00) Region 161 Avg (IOU: 0.583201), count: 1, total_loss = 2.519048 total_bbox = 4340919, rewritten_bbox = 7.680885 %

ghost commented 3 years ago

I have the same symptom with Tesla V100 (32GB).

WongKinYiu commented 3 years ago

image use smaller anchors or lager input resolution or p2 layer prediction or combinations of above.