search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.76k stars 7.96k forks source link

Low loss but mAP = 0.00 ? #849

Closed offchan42 closed 6 years ago

offchan42 commented 6 years ago

After training my model for 900 iterations, I've got loss around 0.04. Very low indeed. But after calculating mAP, it's zero!

Here's the tail output of the training:

897: 0.044502, 0.056504 avg, 0.000647 rate, 1.184289 seconds, 57408 images
Loaded: 0.000045 seconds
Region 16 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000050, .5R: -nan, .75R: -nan,  count: 0
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000065, .5R: -nan, .75R: -nan,  count: 0
Region 16 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000048, .5R: -nan, .75R: -nan,  count: 0
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000061, .5R: -nan, .75R: -nan,  count: 0

 898: 0.000056, 0.050859 avg, 0.000650 rate, 1.185358 seconds, 57472 images
Loaded: 0.000048 seconds
Region 16 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000048, .5R: -nan, .75R: -nan,  count: 0
Region 23 Avg IOU: 0.532580, Class: 0.750525, Obj: 0.001756, No Obj: 0.000057, .5R: 1.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000050, .5R: -nan, .75R: -nan,  count: 0
Region 23 Avg IOU: 0.468856, Class: 0.735366, Obj: 0.001664, No Obj: 0.000060, .5R: 0.500000, .75R: 0.000000,  count: 2

 899: 0.043976, 0.050171 avg, 0.000653 rate, 1.182521 seconds, 57536 images
Loaded: 0.000046 seconds
Region 16 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000046, .5R: -nan, .75R: -nan,  count: 0
Region 23 Avg IOU: 0.398009, Class: 0.598133, Obj: 0.000971, No Obj: 0.000057, .5R: 0.000000, .75R: 0.000000,  count: 1
Region 16 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000049, .5R: -nan, .75R: -nan,  count: 0
Region 23 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000058, .5R: -nan, .75R: -nan,  count: 0

 900: 0.012532, 0.046407 avg, 0.000656 rate, 1.186532 seconds, 57600 images
Saving weights to backup//yolov3-tiny-ch_900.weights
Loaded: 0.000057 seconds
Region 16 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000048, .5R: -nan, .75R: -nan,  count: 0
Region 23 Avg IOU: 0.500981, Class: 0.715004, Obj: 0.002469, No Obj: 0.000055, .5R: 0.500000, .75R: 0.000000,  count: 2
Region 16 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000047, .5R: -nan, .75R: -nan,  count: 0
Region 23 Avg IOU: 0.415602, Class: 0.681575, Obj: 0.002879, No Obj: 0.000056, .5R: 0.000000, .75R: 0.000000,  count: 2

Here's the mAP result:

etections_count = 1, unique_truth_count = 4809
class_id = 0, name = 0,          ap = 0.00 %
class_id = 1, name = 1,          ap = 0.00 %
class_id = 2, name = 2,          ap = 0.00 %
class_id = 3, name = 3,          ap = 0.00 %
class_id = 4, name = 4,          ap = 0.00 %
class_id = 5, name = 5,          ap = 0.00 %
class_id = 6, name = 6,          ap = 0.00 %
class_id = 7, name = 7,          ap = 0.00 %
class_id = 8, name = 8,          ap = 0.00 %
class_id = 9, name = 9,          ap = 0.00 %
class_id = 10, name = char,      ap = 0.00 %
class_id = 11, name = 01-ko-kai,         ap = 0.00 %
class_id = 12, name = 02-kho-khai,       ap = 0.00 %
class_id = 13, name = 03-kho-khuat,      ap = 0.00 %
class_id = 14, name = 04-kho-khwai,      ap = 0.00 %
class_id = 15, name = 05-kho-khon,       ap = 0.00 %
class_id = 16, name = 06-kho-ra-khang,   ap = 0.00 %
class_id = 17, name = 07-ngo-ngu,        ap = 0.00 %
class_id = 18, name = 08-cho-chan,       ap = 0.00 %
class_id = 19, name = 09-cho-ching,      ap = 0.00 %
class_id = 20, name = 10-cho-chang,      ap = 0.00 %
class_id = 21, name = 11-so-so,          ap = 0.00 %
class_id = 22, name = 12-cho-choe,       ap = 0.00 %
class_id = 23, name = 13-yo-ying,        ap = 0.00 %
class_id = 24, name = 14-do-cha-da,      ap = 0.00 %
class_id = 25, name = 15-to-pa-tak,      ap = 0.00 %
class_id = 26, name = 16-tho-than,       ap = 0.00 %
class_id = 27, name = 17-tho-montho,     ap = 0.00 %
class_id = 28, name = 18-tho-phu-thao,   ap = 0.00 %
class_id = 29, name = 19-no-nen,         ap = 0.00 %
class_id = 30, name = 20-do-dek,         ap = 0.00 %
class_id = 31, name = 21-to-tao,         ap = 0.00 %
class_id = 32, name = 22-tho-thung,      ap = 0.00 %
class_id = 33, name = 23-tho-thahan,     ap = 0.00 %
class_id = 34, name = 24-tho-thong,      ap = 0.00 %
class_id = 35, name = 25-no-nu,          ap = 0.00 %
class_id = 36, name = 26-bo-baimai,      ap = 0.00 %
class_id = 37, name = 27-po-pla,         ap = 0.00 %
class_id = 38, name = 28-pho-phueng,     ap = 0.00 %
class_id = 39, name = 29-fo-fa,          ap = 0.00 %
class_id = 40, name = 30-pho-phan,       ap = 0.00 %
class_id = 41, name = 31-fo-fan,         ap = 0.00 %
class_id = 42, name = 32-pho-sam-phao,   ap = 0.00 %
class_id = 43, name = 33-mo-ma,          ap = 0.00 %
class_id = 44, name = 34-yo-yak,         ap = 0.00 %
class_id = 45, name = 35-ro-ruea,        ap = 0.00 %
class_id = 46, name = 36-lo-ling,        ap = 0.00 %
class_id = 47, name = 37-wo-waen,        ap = 0.00 %
class_id = 48, name = 38-so-sala,        ap = 0.00 %
class_id = 49, name = 39-so-rue-si,      ap = 0.00 %
class_id = 50, name = 40-so-suea,        ap = 0.00 %
class_id = 51, name = 41-ho-hip,         ap = 0.00 %
class_id = 52, name = 42-lo-chu-la,      ap = 0.00 %
class_id = 53, name = 43-o-ang,          ap = 0.00 %
class_id = 54, name = 44-ho-nok-huk,     ap = 0.00 %
class_id = 55, name = changwat,          ap = 0.00 %
class_id = 56, name = Amnat-Charoen,     ap = 0.00 %
class_id = 57, name = Ang-Thong,         ap = 0.00 %
class_id = 58, name = Bangkok,   ap = 0.00 %
class_id = 59, name = Bueng-Kan,         ap = 0.00 %
class_id = 60, name = Buri-Ram,          ap = 0.00 %
class_id = 61, name = Chachoengsao,      ap = 0.00 %
class_id = 62, name = Chai-Nat,          ap = 0.00 %
class_id = 63, name = Chaiyaphum,        ap = 0.00 %
class_id = 64, name = Chanthaburi,       ap = 0.00 %
class_id = 65, name = Chiang-Mai,        ap = 0.00 %
class_id = 66, name = Chiang-Rai,        ap = 0.00 %
class_id = 67, name = Chon-Buri,         ap = 0.00 %
class_id = 68, name = Chumphon,          ap = 0.00 %
class_id = 69, name = Kalasin,   ap = 0.00 %
class_id = 70, name = Kamphaeng-Phet,    ap = 0.00 %
class_id = 71, name = Kanchanaburi,      ap = 0.00 %
class_id = 72, name = Khon-Kaen,         ap = 0.00 %
class_id = 73, name = Krabi,     ap = 0.00 %
class_id = 74, name = Lampang,   ap = 0.00 %
class_id = 75, name = Lamphun,   ap = 0.00 %
class_id = 76, name = Loei,      ap = 0.00 %
class_id = 77, name = Lop-Buri,          ap = 0.00 %
class_id = 78, name = Mae-Hong-Son,      ap = 0.00 %
class_id = 79, name = Maha-Sarakham,     ap = 0.00 %
class_id = 80, name = Mukdahan,          ap = 0.00 %
class_id = 81, name = Nakhon-Nayok,      ap = 0.00 %
class_id = 82, name = Nakhon-Pathom,     ap = 0.00 %
class_id = 83, name = Nakhon-Phanom,     ap = 0.00 %
class_id = 84, name = Nakhon-Ratchasima,         ap = 0.00 %
class_id = 85, name = Nakhon-Sawan,      ap = 0.00 %
class_id = 86, name = Nakhon-Si-Thammarat,       ap = 0.00 %
class_id = 87, name = Nan,       ap = 0.00 %
class_id = 88, name = Narathiwat,        ap = 0.00 %
class_id = 89, name = Nong-Bua-Lam-Phu,          ap = 0.00 %
class_id = 90, name = Nong-Khai,         ap = 0.00 %
class_id = 91, name = Nonthaburi,        ap = 0.00 %
class_id = 92, name = Pathum-Thani,      ap = 0.00 %
class_id = 93, name = Pattani,   ap = 0.00 %
class_id = 94, name = Phangnga,          ap = 0.00 %
class_id = 95, name = Phatthalung,       ap = 0.00 %
class_id = 96, name = Phayao,    ap = 0.00 %
class_id = 97, name = Phetchabun,        ap = 0.00 %
class_id = 98, name = Phetchaburi,       ap = 0.00 %
class_id = 99, name = Phichit,   ap = 0.00 %
class_id = 100, name = Phitsanulok,      ap = 0.00 %
class_id = 101, name = Phrae,    ap = 0.00 %
class_id = 102, name = Phra-Nakhon-Si-Ayutthaya,         ap = 0.00 %
class_id = 103, name = Phuket,   ap = 0.00 %
class_id = 104, name = Prachin-Buri,     ap = 0.00 %
class_id = 105, name = Prachuap-Khiri-Khan,      ap = 0.00 %
class_id = 106, name = Ranong,   ap = 0.00 %
class_id = 107, name = Ratchaburi,       ap = 0.00 %
class_id = 108, name = Rayong,   ap = 0.00 %
class_id = 109, name = Roi-Et,   ap = 0.00 %
class_id = 110, name = Sa-Kaeo,          ap = 0.00 %
class_id = 111, name = Sakon-Nakhon,     ap = 0.00 %
class_id = 112, name = Samut-Prakan,     ap = 0.00 %
class_id = 113, name = Samut-Sakhon,     ap = 0.00 %
class_id = 114, name = Samut-Songkhram,          ap = 0.00 %
class_id = 115, name = Saraburi,         ap = 0.00 %
class_id = 116, name = Satun,    ap = 0.00 %
class_id = 117, name = Sing-Buri,        ap = 0.00 %
class_id = 118, name = Si-Sa-Ket,        ap = 0.00 %
class_id = 119, name = Songkhla,         ap = 0.00 %
class_id = 120, name = Sukhothai,        ap = 0.00 %
class_id = 121, name = Suphan-Buri,      ap = 0.00 %
class_id = 122, name = Surat-Thani,      ap = 0.00 %
class_id = 123, name = Surin,    ap = 0.00 %
class_id = 124, name = Tak,      ap = 0.00 %
class_id = 125, name = Trang,    ap = 0.00 %
class_id = 126, name = Trat,     ap = 0.00 %
class_id = 127, name = Ubon-Ratchathani,         ap = 0.00 %
class_id = 128, name = Udon-Thani,       ap = 0.00 %
class_id = 129, name = Uthai-Thani,      ap = 0.00 %
class_id = 130, name = Uttaradit,        ap = 0.00 %
class_id = 131, name = Yala,     ap = 0.00 %
class_id = 132, name = Yasothon,         ap = 0.00 %
 for thresh = 0.25, precision = -nan, recall = 0.00, F1-score = -nan
 for thresh = 0.25, TP = 0, FP = 0, FN = 4809, average IoU = 0.00 %

 mean average precision (mAP) = 0.000000, or 0.00 %
Total Detection Time: 7.000000 Seconds

I want to ask, why is it possible that the mAP is still reporting zero? And also if AVG IOU is nan when training, is it fine?

offchan42 commented 6 years ago

After training for more 100 iterations, the map is now 0.000684 which is roughly 0.07% Is it possible that this just needs more training time and I'm just being panic that it's nowhere near what I want? I'm expecting that at least the mAP value should be high enough by now. This training is so tricky. I trained for more 200 iterations and mAP is reduced to 0.01% again. I don't know what to expect here. Please suggest me something based on your experience. Do I need to generate those anchors thing? I don't know what it is and I don't know whether or not it's necessary.

Also, a question, what is the number before the word rate?

AlexeyAB commented 6 years ago

Your dataset is wrong: https://github.com/AlexeyAB/darknet/issues/847#issuecomment-390318365

offchan42 commented 6 years ago

Thank you!