search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.75k stars 7.96k forks source link

Fail to train YOLO #1020

Open ido-ran opened 6 years ago

ido-ran commented 6 years ago

Hi, I'm trying to train YOLO to detect soy milk bottle. I've follow your instructions but when I try to run the detection I'm getting

Assertion failed: (n != 0 && x_gpu.len - x_gpu.off == n), function opencl_pull_array, file /Users/idoran/Dev/Walmart/object-detection-poc/sowson-darknet/src/opencl.c, line 797.
Abort trap: 6

I am using a build of darknet which is optimized for Mac but I think the code should work with the same data. I am able to detect using yolo9000 weights without a problem.

You can see all of the configuration I've used here https://gist.github.com/ido-ran/8ff0de8a09ae3b1815f31f295666911e

I'm not sure it means anything but if I run ./darknet detector valid cfg/soy-milk.data cfg/yolov3-tiny-soy-milk.cfg yolov3-tiny.conv.15 it also fails with the same error.

AlexeyAB commented 6 years ago

Hi,

Perhaps bug in this repository: https://github.com/sowson/darknet

What GPU do you use?

ido-ran commented 6 years ago

This is very possible, I'm saying it because when I run the train command sometime it works but sometime it crashes, after I re-run the command it is working again. I'm using Radeon Pro 560 4096 MB on MacBook Pro 14,3.

ido-ran commented 6 years ago

I've run your code on EC2 Deep Learning Ubuntu on p2.xlarge instance. I've copy the exact same configuration I've used on my Mac and now it's running. This is the output so far (see bellow).

Two questions:

  1. I see there are lots of -nan in Avg IOU - is that an indication that I'm doing something wrong? Also the avg loss is raising - again am I doing something wrong?

  2. Last, I've started with 30 pictures of the objects. From what I understood it usually takes at least 100 to get good detection - will I be able to get anything with 30?

Thank you very much, Ido.

$ ./darknet detector train cfg/soy-milk.data cfg/yolov3-tiny-soy-milk.cfg yolov3-tiny.conv.15 
yolov3-tiny-soy-milk
layer     filters    size              input                output
   0 conv     16  3 x 3 / 1   416 x 416 x   3   ->   416 x 416 x  16 0.150 BF
   1 max          2 x 2 / 2   416 x 416 x  16   ->   208 x 208 x  16 0.003 BF
   2 conv     32  3 x 3 / 1   208 x 208 x  16   ->   208 x 208 x  32 0.399 BF
   3 max          2 x 2 / 2   208 x 208 x  32   ->   104 x 104 x  32 0.001 BF
   4 conv     64  3 x 3 / 1   104 x 104 x  32   ->   104 x 104 x  64 0.399 BF
   5 max          2 x 2 / 2   104 x 104 x  64   ->    52 x  52 x  64 0.001 BF
   6 conv    128  3 x 3 / 1    52 x  52 x  64   ->    52 x  52 x 128 0.399 BF
   7 max          2 x 2 / 2    52 x  52 x 128   ->    26 x  26 x 128 0.000 BF
   8 conv    256  3 x 3 / 1    26 x  26 x 128   ->    26 x  26 x 256 0.399 BF
   9 max          2 x 2 / 2    26 x  26 x 256   ->    13 x  13 x 256 0.000 BF
  10 conv    512  3 x 3 / 1    13 x  13 x 256   ->    13 x  13 x 512 0.399 BF
  11 max          2 x 2 / 1    13 x  13 x 512   ->    13 x  13 x 512 0.000 BF
  12 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024 1.595 BF
  13 conv    256  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 256 0.089 BF
  14 conv    512  3 x 3 / 1    13 x  13 x 256   ->    13 x  13 x 512 0.399 BF
  15 conv     18  1 x 1 / 1    13 x  13 x 512   ->    13 x  13 x  18 0.003 BF
  16 yolo
  17 route  13
  18 conv    128  1 x 1 / 1    13 x  13 x 256   ->    13 x  13 x 128 0.011 BF
  19 upsample            2x    13 x  13 x 128   ->    26 x  26 x 128
  20 route  19 8
  21 conv    256  3 x 3 / 1    26 x  26 x 384   ->    26 x  26 x 256 1.196 BF
  22 conv     18  1 x 1 / 1    26 x  26 x 256   ->    26 x  26 x  18 0.006 BF
  23 yolo
Total BFLOPS 5.448 
Loading weights from yolov3-tiny.conv.15...
 seen 32 
Done!
Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005
Resizing
480 x 480 
 try to allocate workspace = 12267521 * sizeof(float),  CUDA allocate done! 
Loaded: 6.841417 seconds
Region 16 Avg IOU: 0.407115, Class: 0.478230, Obj: 0.457788, No Obj: 0.479276, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555411, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.452558, Class: 0.500705, Obj: 0.524700, No Obj: 0.479769, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555603, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.444210, Class: 0.508083, Obj: 0.441740, No Obj: 0.480450, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555032, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.429801, Class: 0.600578, Obj: 0.466166, No Obj: 0.480324, .5R: 0.500000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555397, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.417630, Class: 0.488389, Obj: 0.422469, No Obj: 0.481849, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555326, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.368919, Class: 0.596879, Obj: 0.465340, No Obj: 0.479702, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555318, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.317168, Class: 0.484774, Obj: 0.478646, No Obj: 0.479928, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554974, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.399225, Class: 0.521759, Obj: 0.535160, No Obj: 0.481311, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555697, .5R: -nan, .75R: -nan,  count: 0

 1: 519.712341, 519.712341 avg loss, 0.000000 rate, 2.142625 seconds, 64 images
Loaded: 6.468971 seconds
Region 16 Avg IOU: 0.334358, Class: 0.531611, Obj: 0.504255, No Obj: 0.482287, .5R: 0.125000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555749, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.440310, Class: 0.555462, Obj: 0.514824, No Obj: 0.481093, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554958, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.406594, Class: 0.549988, Obj: 0.424131, No Obj: 0.480106, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555526, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.412686, Class: 0.536605, Obj: 0.440484, No Obj: 0.480916, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555282, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.352932, Class: 0.542724, Obj: 0.412212, No Obj: 0.478534, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555582, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.493763, Class: 0.498229, Obj: 0.508266, No Obj: 0.480397, .5R: 0.375000, .75R: 0.250000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555418, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.504841, Class: 0.516375, Obj: 0.520536, No Obj: 0.480559, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555275, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.414461, Class: 0.479262, Obj: 0.486689, No Obj: 0.480635, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555533, .5R: -nan, .75R: -nan,  count: 0

 2: 519.797913, 519.720886 avg loss, 0.000000 rate, 2.054143 seconds, 128 images
Loaded: 6.946068 seconds
Region 16 Avg IOU: 0.354932, Class: 0.538129, Obj: 0.469295, No Obj: 0.478171, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555424, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.310553, Class: 0.491090, Obj: 0.495223, No Obj: 0.480587, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555416, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.411491, Class: 0.507312, Obj: 0.484129, No Obj: 0.478989, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555474, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.421734, Class: 0.435217, Obj: 0.522451, No Obj: 0.481129, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554969, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.461163, Class: 0.459436, Obj: 0.522738, No Obj: 0.480389, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555480, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.326951, Class: 0.515533, Obj: 0.450851, No Obj: 0.479567, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555350, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.310973, Class: 0.491075, Obj: 0.482859, No Obj: 0.481843, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555572, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.510858, Class: 0.535261, Obj: 0.498287, No Obj: 0.480250, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555199, .5R: -nan, .75R: -nan,  count: 0

 3: 519.657654, 519.714539 avg loss, 0.000000 rate, 1.982708 seconds, 192 images
Loaded: 6.590878 seconds
Region 16 Avg IOU: 0.574990, Class: 0.578249, Obj: 0.370807, No Obj: 0.478876, .5R: 0.625000, .75R: 0.375000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555519, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.378522, Class: 0.494125, Obj: 0.438442, No Obj: 0.481509, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555157, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.401013, Class: 0.505681, Obj: 0.439781, No Obj: 0.480441, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554512, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.481850, Class: 0.506944, Obj: 0.477520, No Obj: 0.480515, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555776, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.351952, Class: 0.502761, Obj: 0.561409, No Obj: 0.479939, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555551, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.441045, Class: 0.520838, Obj: 0.397308, No Obj: 0.480010, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555267, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.331337, Class: 0.580544, Obj: 0.452335, No Obj: 0.481238, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554899, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.402755, Class: 0.542475, Obj: 0.515033, No Obj: 0.479277, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555592, .5R: -nan, .75R: -nan,  count: 0

 4: 519.254089, 519.668518 avg loss, 0.000000 rate, 1.988151 seconds, 256 images
Loaded: 6.709271 seconds
Region 16 Avg IOU: 0.506771, Class: 0.500657, Obj: 0.475075, No Obj: 0.481084, .5R: 0.500000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554952, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.417597, Class: 0.531553, Obj: 0.387895, No Obj: 0.481488, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555563, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.422338, Class: 0.543080, Obj: 0.517493, No Obj: 0.480809, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555674, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.492900, Class: 0.541299, Obj: 0.505664, No Obj: 0.480792, .5R: 0.625000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555498, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.420013, Class: 0.483816, Obj: 0.452512, No Obj: 0.479133, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555180, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.300954, Class: 0.414409, Obj: 0.417571, No Obj: 0.481446, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555683, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.425695, Class: 0.558788, Obj: 0.523720, No Obj: 0.481512, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555430, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.350482, Class: 0.547843, Obj: 0.438915, No Obj: 0.481596, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555291, .5R: -nan, .75R: -nan,  count: 0

 5: 520.037415, 519.705383 avg loss, 0.000000 rate, 2.022538 seconds, 320 images
Loaded: 6.970871 seconds
Region 16 Avg IOU: 0.319514, Class: 0.455323, Obj: 0.447522, No Obj: 0.480969, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555569, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.469279, Class: 0.466189, Obj: 0.402951, No Obj: 0.478696, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555289, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.391271, Class: 0.584334, Obj: 0.416512, No Obj: 0.479975, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554671, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.335921, Class: 0.567953, Obj: 0.431346, No Obj: 0.480599, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555976, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.434304, Class: 0.516697, Obj: 0.407717, No Obj: 0.479140, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555605, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.214486, Class: 0.570911, Obj: 0.396307, No Obj: 0.479730, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554820, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.344997, Class: 0.508740, Obj: 0.455776, No Obj: 0.480372, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554949, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.373442, Class: 0.553314, Obj: 0.504749, No Obj: 0.480009, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554955, .5R: -nan, .75R: -nan,  count: 0

 6: 519.518555, 519.686707 avg loss, 0.000000 rate, 1.965215 seconds, 384 images
Loaded: 6.948162 seconds
Region 16 Avg IOU: 0.505475, Class: 0.547545, Obj: 0.372827, No Obj: 0.479719, .5R: 0.500000, .75R: 0.250000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555738, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.345716, Class: 0.526023, Obj: 0.457787, No Obj: 0.479524, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554846, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.408628, Class: 0.525723, Obj: 0.506875, No Obj: 0.480121, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555492, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.381077, Class: 0.539805, Obj: 0.480885, No Obj: 0.479642, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555150, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.438250, Class: 0.452409, Obj: 0.416514, No Obj: 0.480923, .5R: 0.250000, .75R: 0.250000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555143, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.331294, Class: 0.549649, Obj: 0.468036, No Obj: 0.481632, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555474, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.398288, Class: 0.459366, Obj: 0.389947, No Obj: 0.479637, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555708, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.353954, Class: 0.461746, Obj: 0.566367, No Obj: 0.479530, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555663, .5R: -nan, .75R: -nan,  count: 0

 7: 519.675049, 519.685547 avg loss, 0.000000 rate, 2.194135 seconds, 448 images
Loaded: 6.163149 seconds
Region 16 Avg IOU: 0.362121, Class: 0.535789, Obj: 0.464166, No Obj: 0.479523, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555392, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.335721, Class: 0.563832, Obj: 0.436869, No Obj: 0.480377, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554784, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.519785, Class: 0.529793, Obj: 0.461230, No Obj: 0.480556, .5R: 0.500000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555101, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.468607, Class: 0.559354, Obj: 0.497471, No Obj: 0.480113, .5R: 0.375000, .75R: 0.250000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556274, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.500249, Class: 0.487789, Obj: 0.517975, No Obj: 0.479251, .5R: 0.625000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556275, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.469798, Class: 0.518885, Obj: 0.481644, No Obj: 0.478734, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555381, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.331246, Class: 0.555743, Obj: 0.433918, No Obj: 0.480062, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554932, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.497367, Class: 0.543190, Obj: 0.406061, No Obj: 0.480816, .5R: 0.375000, .75R: 0.250000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555642, .5R: -nan, .75R: -nan,  count: 0

 8: 519.639771, 519.680969 avg loss, 0.000000 rate, 2.061149 seconds, 512 images
Loaded: 7.184076 seconds
Region 16 Avg IOU: 0.328165, Class: 0.551984, Obj: 0.390409, No Obj: 0.481193, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555392, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.309660, Class: 0.463748, Obj: 0.423120, No Obj: 0.480694, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554890, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.460607, Class: 0.406122, Obj: 0.410410, No Obj: 0.480126, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555467, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.392083, Class: 0.534464, Obj: 0.442172, No Obj: 0.479006, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555040, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.425860, Class: 0.510454, Obj: 0.454224, No Obj: 0.479662, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554964, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.433464, Class: 0.525874, Obj: 0.415086, No Obj: 0.480291, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555491, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.409714, Class: 0.510207, Obj: 0.394024, No Obj: 0.480676, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555299, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.404582, Class: 0.545243, Obj: 0.413779, No Obj: 0.480385, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555541, .5R: -nan, .75R: -nan,  count: 0

 9: 519.485046, 519.661377 avg loss, 0.000000 rate, 1.994634 seconds, 576 images
Loaded: 6.747834 seconds
Region 16 Avg IOU: 0.511895, Class: 0.520848, Obj: 0.379351, No Obj: 0.480059, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555621, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.312503, Class: 0.469639, Obj: 0.491274, No Obj: 0.479595, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555259, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.475522, Class: 0.437696, Obj: 0.527687, No Obj: 0.481076, .5R: 0.500000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555386, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.353377, Class: 0.490551, Obj: 0.490010, No Obj: 0.480227, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555141, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.406649, Class: 0.592424, Obj: 0.413151, No Obj: 0.479591, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555132, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.342488, Class: 0.510995, Obj: 0.454608, No Obj: 0.481268, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555390, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.475793, Class: 0.481350, Obj: 0.478190, No Obj: 0.479787, .5R: 0.375000, .75R: 0.250000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556005, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.381995, Class: 0.514362, Obj: 0.519753, No Obj: 0.480445, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554513, .5R: -nan, .75R: -nan,  count: 0

 10: 519.791809, 519.674438 avg loss, 0.000000 rate, 1.900327 seconds, 640 images
Resizing
320 x 320 
 try to allocate workspace = 3911681 * sizeof(float),  CUDA allocate done! 
Loaded: 8.078530 seconds
Region 16 Avg IOU: 0.363938, Class: 0.527672, Obj: 0.405312, No Obj: 0.480006, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555354, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.414004, Class: 0.549741, Obj: 0.506182, No Obj: 0.479037, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555771, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.473051, Class: 0.516233, Obj: 0.443256, No Obj: 0.480103, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555351, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.455295, Class: 0.595991, Obj: 0.403857, No Obj: 0.479477, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555940, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.436232, Class: 0.536756, Obj: 0.428919, No Obj: 0.478272, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555974, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.381061, Class: 0.459227, Obj: 0.442673, No Obj: 0.478557, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555596, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.320694, Class: 0.569192, Obj: 0.445505, No Obj: 0.477539, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: 0.138529, Class: 0.416160, Obj: 0.558242, No Obj: 0.555449, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.426474, Class: 0.537094, Obj: 0.354402, No Obj: 0.479526, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554709, .5R: -nan, .75R: -nan,  count: 0

 11: 232.243454, 490.931335 avg loss, 0.000000 rate, 0.953717 seconds, 704 images
Loaded: 7.512951 seconds
Region 16 Avg IOU: 0.271581, Class: 0.478328, Obj: 0.435040, No Obj: 0.478505, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556013, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.421777, Class: 0.581072, Obj: 0.490102, No Obj: 0.479098, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555842, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.534457, Class: 0.530394, Obj: 0.364951, No Obj: 0.480134, .5R: 0.625000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556076, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.379465, Class: 0.503803, Obj: 0.345234, No Obj: 0.479824, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555828, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.293277, Class: 0.484992, Obj: 0.434417, No Obj: 0.478036, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555898, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.355637, Class: 0.548891, Obj: 0.424046, No Obj: 0.479535, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554737, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.284924, Class: 0.488430, Obj: 0.386537, No Obj: 0.479950, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554861, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.363545, Class: 0.540376, Obj: 0.352645, No Obj: 0.478127, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: 0.242141, Class: 0.541179, Obj: 0.504431, No Obj: 0.555954, .5R: 0.000000, .75R: 0.000000,  count: 1

 12: 232.862701, 465.124481 avg loss, 0.000000 rate, 1.049934 seconds, 768 images
Loaded: 7.282682 seconds
Region 16 Avg IOU: 0.428034, Class: 0.495781, Obj: 0.465584, No Obj: 0.477743, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555377, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.313716, Class: 0.492464, Obj: 0.403746, No Obj: 0.480834, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554889, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.362134, Class: 0.635697, Obj: 0.361484, No Obj: 0.478433, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555893, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.410704, Class: 0.432639, Obj: 0.465616, No Obj: 0.480013, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555949, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.478294, Class: 0.539578, Obj: 0.404584, No Obj: 0.478990, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556280, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.408040, Class: 0.508827, Obj: 0.420263, No Obj: 0.480140, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: 0.571518, Class: 0.686171, Obj: 0.649063, No Obj: 0.556473, .5R: 1.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.500261, Class: 0.496047, Obj: 0.436126, No Obj: 0.480290, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555895, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.397791, Class: 0.452046, Obj: 0.421752, No Obj: 0.481061, .5R: 0.250000, .75R: 0.250000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555904, .5R: -nan, .75R: -nan,  count: 0

 13: 232.564667, 441.868500 avg loss, 0.000000 rate, 1.063896 seconds, 832 images
Loaded: 7.138266 seconds
Region 16 Avg IOU: 0.498969, Class: 0.514850, Obj: 0.529809, No Obj: 0.482243, .5R: 0.625000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555329, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.345454, Class: 0.525983, Obj: 0.416044, No Obj: 0.479766, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555104, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.416337, Class: 0.538848, Obj: 0.403364, No Obj: 0.478724, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555322, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.422527, Class: 0.521790, Obj: 0.438273, No Obj: 0.477868, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555620, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.378867, Class: 0.544358, Obj: 0.428213, No Obj: 0.478991, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555713, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.407152, Class: 0.498571, Obj: 0.423191, No Obj: 0.480097, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555732, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.480355, Class: 0.573245, Obj: 0.428066, No Obj: 0.480405, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555318, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.380296, Class: 0.480527, Obj: 0.419170, No Obj: 0.479850, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555282, .5R: -nan, .75R: -nan,  count: 0

 14: 232.125214, 420.894165 avg loss, 0.000000 rate, 0.889522 seconds, 896 images
Loaded: 7.100080 seconds
Region 16 Avg IOU: 0.401084, Class: 0.528679, Obj: 0.472249, No Obj: 0.479551, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556395, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.401579, Class: 0.536202, Obj: 0.449629, No Obj: 0.479411, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556002, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.405463, Class: 0.614349, Obj: 0.398393, No Obj: 0.479816, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555398, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.321565, Class: 0.430946, Obj: 0.390926, No Obj: 0.477768, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555590, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.367263, Class: 0.555362, Obj: 0.418316, No Obj: 0.478481, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: 0.228820, Class: 0.325771, Obj: 0.662389, No Obj: 0.555215, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.426864, Class: 0.467016, Obj: 0.410882, No Obj: 0.479835, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556015, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.437181, Class: 0.528377, Obj: 0.523479, No Obj: 0.479517, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556073, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.402108, Class: 0.519907, Obj: 0.391446, No Obj: 0.478666, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556282, .5R: -nan, .75R: -nan,  count: 0

 15: 232.584930, 402.063232 avg loss, 0.000000 rate, 0.919596 seconds, 960 images
Loaded: 7.346638 seconds
Region 16 Avg IOU: 0.365231, Class: 0.670736, Obj: 0.384962, No Obj: 0.478540, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555237, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.428947, Class: 0.503838, Obj: 0.364494, No Obj: 0.480119, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556622, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.290538, Class: 0.489365, Obj: 0.427048, No Obj: 0.479499, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554901, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.384477, Class: 0.584612, Obj: 0.381891, No Obj: 0.479778, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555309, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.338209, Class: 0.538503, Obj: 0.383764, No Obj: 0.478083, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556095, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.436006, Class: 0.449177, Obj: 0.483631, No Obj: 0.480482, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555909, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.346305, Class: 0.613294, Obj: 0.365031, No Obj: 0.479909, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556128, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.380401, Class: 0.533620, Obj: 0.406005, No Obj: 0.479672, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555202, .5R: -nan, .75R: -nan,  count: 0

 16: 232.715469, 385.128448 avg loss, 0.000000 rate, 0.986074 seconds, 1024 images
Loaded: 7.467689 seconds
Region 16 Avg IOU: 0.317804, Class: 0.618536, Obj: 0.457491, No Obj: 0.479031, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555808, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.447534, Class: 0.552368, Obj: 0.486065, No Obj: 0.478911, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555704, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.320338, Class: 0.505086, Obj: 0.426282, No Obj: 0.477560, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555773, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.437767, Class: 0.591908, Obj: 0.422783, No Obj: 0.477736, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555569, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.450480, Class: 0.546024, Obj: 0.321785, No Obj: 0.479465, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556549, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.415289, Class: 0.523268, Obj: 0.438379, No Obj: 0.478999, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555669, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.465323, Class: 0.484784, Obj: 0.415824, No Obj: 0.480028, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556138, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.470498, Class: 0.538995, Obj: 0.395725, No Obj: 0.477535, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555314, .5R: -nan, .75R: -nan,  count: 0

 17: 232.265015, 369.842102 avg loss, 0.000000 rate, 0.996753 seconds, 1088 images
Loaded: 7.327591 seconds
Region 16 Avg IOU: 0.390967, Class: 0.517481, Obj: 0.446900, No Obj: 0.480288, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555947, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.349739, Class: 0.573750, Obj: 0.446552, No Obj: 0.478474, .5R: 0.125000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555563, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.564836, Class: 0.560503, Obj: 0.351342, No Obj: 0.478606, .5R: 0.625000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555308, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.425660, Class: 0.587814, Obj: 0.437029, No Obj: 0.479006, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555645, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.461140, Class: 0.554477, Obj: 0.421360, No Obj: 0.478866, .5R: 0.625000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555420, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.473382, Class: 0.513054, Obj: 0.429644, No Obj: 0.479776, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555659, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.336206, Class: 0.513795, Obj: 0.360913, No Obj: 0.477961, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555663, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.378293, Class: 0.484867, Obj: 0.396036, No Obj: 0.479833, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555847, .5R: -nan, .75R: -nan,  count: 0

 18: 232.448715, 356.102753 avg loss, 0.000000 rate, 1.047342 seconds, 1152 images
Loaded: 7.022127 seconds
Region 16 Avg IOU: 0.380184, Class: 0.575242, Obj: 0.391572, No Obj: 0.478015, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555635, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.262899, Class: 0.524945, Obj: 0.413804, No Obj: 0.480973, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555576, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.454871, Class: 0.565486, Obj: 0.383808, No Obj: 0.478375, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555758, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.384986, Class: 0.544711, Obj: 0.359297, No Obj: 0.478940, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556141, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.391684, Class: 0.536682, Obj: 0.363327, No Obj: 0.479317, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555657, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.395831, Class: 0.517229, Obj: 0.397898, No Obj: 0.481382, .5R: 0.125000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555862, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.410974, Class: 0.435690, Obj: 0.447492, No Obj: 0.478636, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: 0.569860, Class: 0.661778, Obj: 0.490645, No Obj: 0.555680, .5R: 1.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.347652, Class: 0.549717, Obj: 0.444924, No Obj: 0.477798, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555064, .5R: -nan, .75R: -nan,  count: 0

 19: 232.556717, 343.748138 avg loss, 0.000000 rate, 0.964266 seconds, 1216 images
Loaded: 7.189559 seconds
Region 16 Avg IOU: 0.320156, Class: 0.584257, Obj: 0.360122, No Obj: 0.479583, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: 0.290840, Class: 0.657232, Obj: 0.637049, No Obj: 0.555738, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: 0.533410, Class: 0.556942, Obj: 0.388271, No Obj: 0.478848, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556596, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.494999, Class: 0.537369, Obj: 0.386189, No Obj: 0.477549, .5R: 0.625000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556099, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.352096, Class: 0.468002, Obj: 0.454937, No Obj: 0.479200, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555394, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.378430, Class: 0.575958, Obj: 0.366737, No Obj: 0.478805, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555382, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.480959, Class: 0.499323, Obj: 0.518912, No Obj: 0.478850, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555764, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.372707, Class: 0.513107, Obj: 0.429788, No Obj: 0.479061, .5R: 0.125000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556075, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.331021, Class: 0.509301, Obj: 0.445481, No Obj: 0.478502, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555923, .5R: -nan, .75R: -nan,  count: 0

 20: 232.692429, 332.642578 avg loss, 0.000000 rate, 0.989309 seconds, 1280 images
Resizing
320 x 320 
 try to allocate workspace = 3911681 * sizeof(float),  CUDA allocate done! 
Loaded: 8.114766 seconds
Region 16 Avg IOU: 0.281614, Class: 0.475561, Obj: 0.476648, No Obj: 0.478684, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555779, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.366992, Class: 0.527137, Obj: 0.375555, No Obj: 0.479075, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555753, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.444465, Class: 0.643574, Obj: 0.420435, No Obj: 0.479106, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555133, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.425894, Class: 0.583094, Obj: 0.432783, No Obj: 0.478526, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555787, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.460489, Class: 0.540978, Obj: 0.376405, No Obj: 0.479675, .5R: 0.750000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556088, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.403410, Class: 0.548763, Obj: 0.388167, No Obj: 0.478179, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555899, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.370496, Class: 0.574330, Obj: 0.435567, No Obj: 0.478231, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555616, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.456513, Class: 0.495127, Obj: 0.411777, No Obj: 0.480407, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556482, .5R: -nan, .75R: -nan,  count: 0

 21: 232.429321, 322.621246 avg loss, 0.000000 rate, 0.959692 seconds, 1344 images
Loaded: 7.030413 seconds
Region 16 Avg IOU: 0.302565, Class: 0.547527, Obj: 0.465888, No Obj: 0.479130, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555826, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.478976, Class: 0.486582, Obj: 0.382970, No Obj: 0.478249, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556003, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.398850, Class: 0.546715, Obj: 0.457738, No Obj: 0.480061, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555602, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.291144, Class: 0.517740, Obj: 0.429005, No Obj: 0.479133, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555855, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.513247, Class: 0.547142, Obj: 0.405596, No Obj: 0.478595, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555950, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.502268, Class: 0.456541, Obj: 0.354037, No Obj: 0.479998, .5R: 0.625000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555655, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.318662, Class: 0.530392, Obj: 0.409051, No Obj: 0.478661, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555531, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.394811, Class: 0.552112, Obj: 0.347899, No Obj: 0.480009, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555829, .5R: -nan, .75R: -nan,  count: 0

 22: 232.699219, 313.629059 avg loss, 0.000000 rate, 0.927866 seconds, 1408 images
Loaded: 7.011241 seconds
Region 16 Avg IOU: 0.453065, Class: 0.453714, Obj: 0.421547, No Obj: 0.479755, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555669, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.444302, Class: 0.526435, Obj: 0.450028, No Obj: 0.478088, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555834, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.374945, Class: 0.461781, Obj: 0.425279, No Obj: 0.479216, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556164, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.408583, Class: 0.581694, Obj: 0.436075, No Obj: 0.478318, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555843, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.430173, Class: 0.501919, Obj: 0.417013, No Obj: 0.479323, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555760, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.344899, Class: 0.576337, Obj: 0.425410, No Obj: 0.479001, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555379, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.332694, Class: 0.506131, Obj: 0.321977, No Obj: 0.477860, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555972, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.330807, Class: 0.521942, Obj: 0.457205, No Obj: 0.479135, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554886, .5R: -nan, .75R: -nan,  count: 0

 23: 232.715027, 305.537659 avg loss, 0.000000 rate, 1.179729 seconds, 1472 images
Loaded: 6.917263 seconds
Region 16 Avg IOU: 0.402553, Class: 0.466380, Obj: 0.444798, No Obj: 0.479090, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555888, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.333876, Class: 0.557396, Obj: 0.397338, No Obj: 0.479385, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555361, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.362563, Class: 0.523760, Obj: 0.436113, No Obj: 0.479833, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555827, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.481127, Class: 0.583766, Obj: 0.333071, No Obj: 0.480509, .5R: 0.375000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555788, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.333005, Class: 0.599728, Obj: 0.386864, No Obj: 0.479584, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555177, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.419236, Class: 0.522071, Obj: 0.428986, No Obj: 0.478310, .5R: 0.500000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555487, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.302523, Class: 0.578635, Obj: 0.366460, No Obj: 0.479614, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555533, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.405565, Class: 0.462603, Obj: 0.413134, No Obj: 0.479788, .5R: 0.125000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556155, .5R: -nan, .75R: -nan,  count: 0

 24: 232.563187, 298.240204 avg loss, 0.000000 rate, 0.903927 seconds, 1536 images
Loaded: 7.309346 seconds
Region 16 Avg IOU: 0.394126, Class: 0.509595, Obj: 0.380120, No Obj: 0.478795, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556418, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.271995, Class: 0.540265, Obj: 0.322754, No Obj: 0.479207, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554968, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.308236, Class: 0.535645, Obj: 0.440182, No Obj: 0.477303, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556223, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.381319, Class: 0.527324, Obj: 0.460543, No Obj: 0.479277, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555979, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.361125, Class: 0.507841, Obj: 0.469890, No Obj: 0.479534, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555973, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.575528, Class: 0.561606, Obj: 0.455172, No Obj: 0.479418, .5R: 0.750000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555941, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.373367, Class: 0.536108, Obj: 0.416714, No Obj: 0.478222, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555101, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.474925, Class: 0.496088, Obj: 0.370632, No Obj: 0.477123, .5R: 0.500000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.554943, .5R: -nan, .75R: -nan,  count: 0

 25: 232.391693, 291.655365 avg loss, 0.000000 rate, 0.980616 seconds, 1600 images
Loaded: 7.583517 seconds
Region 16 Avg IOU: 0.365047, Class: 0.494170, Obj: 0.538386, No Obj: 0.477869, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555737, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.464235, Class: 0.474843, Obj: 0.398423, No Obj: 0.478767, .5R: 0.250000, .75R: 0.125000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555331, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.436422, Class: 0.497376, Obj: 0.358240, No Obj: 0.478889, .5R: 0.375000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555233, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.458171, Class: 0.548639, Obj: 0.339580, No Obj: 0.479376, .5R: 0.500000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.556628, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.292503, Class: 0.523010, Obj: 0.411699, No Obj: 0.479938, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555437, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.331203, Class: 0.535477, Obj: 0.433378, No Obj: 0.480428, .5R: 0.000000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555655, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.395645, Class: 0.566657, Obj: 0.436124, No Obj: 0.479100, .5R: 0.125000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555664, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: 0.419484, Class: 0.516889, Obj: 0.358655, No Obj: 0.479249, .5R: 0.250000, .75R: 0.000000,  count: 8
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.555748, .5R: -nan, .75R: -nan,  count: 0

 26: 232.405746, 285.730408 avg loss, 0.000000 rate, 0.980236 seconds, 1664 images
L
AlexeyAB commented 6 years ago

  1. If nan isn't in avg-los - then this is normal. I don't see that avg-loss is increased in your log.

  2. You will able to detect object on trained images. For production preferably to use 2000 images per class or more.