search

pjreddie / darknet

Convolutional Neural Networks
http://pjreddie.com/darknet/
Other
25.82k stars 21.32k forks source link

map with different results #2518

Open beasnap opened 2 years ago

beasnap commented 2 years ago

hello, everyone

I have a question.

Why is the result after training different from the result using the map function?

  1. Results displayed after learning
  2. Result of using map option in detector (ex. darknet.exe detector map xxx.cfg xxx.weights -iou_thresh 0.5)

model : yolov3-tiny

  1. Contents CUDA allocate done!

    calculation mAP (mean average precision)... Detection layer: 16 - type = 28 Detection layer: 23 - type = 28 64 detections_count = 3770, unique_truth_count = 233 class_id = 0, name = normal, ap = 3.12% (TP = 0, FP = 0) class_id = 1, name = abnormal, ap = 86.71% (TP = 90, FP = 10)

    for conf_thresh = 0.25, precision = 0.90, recall = 0.39, F1-score = 0.54 for conf_thresh = 0.25, TP = 90, FP = 10, FN = 143, average IoU = 73.33 %

    IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.449152, or 44.92 % Total Detection Time: 0 Seconds

Set -points flag: -points 101 for MS COCO -points 11 for PascalVOC 2007 (uncomment difficult in voc.data) -points 0 (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset

mean_average_precision (mAP@0.5) = 0.449152 Saving weights to backup//yolov3-tiny-solvit_2_classes_640x640_4000.weights Saving weights to backup//yolov3-tiny-solvit_2_classes_640x640_last.weights Saving weights to backup//yolov3-tiny-solvit_2_classes_640x640_final.weights If you want to train from the beginning, then use flag in the end of training command: -clear

  1. Contents Done! Loaded 24 layers from weights-file

    calculation mAP (mean average precision)... Detection layer: 16 - type = 28 Detection layer: 23 - type = 28 64 detections_count = 3968, unique_truth_count = 233 class_id = 0, name = normal, ap = 6.53% (TP = 3, FP = 25) class_id = 1, name = abnormal, ap = 85.76% (TP = 92, FP = 11)

    for conf_thresh = 0.25, precision = 0.73, recall = 0.41, F1-score = 0.52 for conf_thresh = 0.25, TP = 95, FP = 36, FN = 138, average IoU = 61.09 %

    IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.461492, or 46.15 % Total Detection Time: 1 Seconds

Set -points flag: -points 101 for MS COCO -points 11 for PascalVOC 2007 (uncomment difficult in voc.data) -points 0 (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset

Anyone please help me!!

DieBirke commented 2 years ago

Hello,

I have the same problem. It definately has something to with the detections_count as these vary drasticly in my runs and the Precision does too. I don't know why they are different and what exactly they mean. Is there a way to set these values? I could not find any infos regarding detection_coint, unique_truth_count or ranks.

First time using map:

detections_count = 2018, unique_truth_count = 107  
class_id = 0, name = coa, ap = 62.99%        (TP = 60, FP = 29) 
class_id = 1, name = knight, ap = 100.00%        (TP = 1, FP = 0) 

 for conf_thresh = 0.25, precision = 0.68, recall = 0.57, F1-score = 0.62 
 for conf_thresh = 0.25, TP = 61, FP = 29, FN = 46, average IoU = 49.14 % 

 IoU threshold = 50 %, used Area-Under-Curve for each unique Recall 
 mean average precision (mAP@0.50) = 0.814938, or 81.49 %

Second time using Map

detections_count = 216, unique_truth_count = 107  
 rank = 0 of ranks = 216 
 rank = 100 of ranks = 216 
 rank = 200 of ranks = 216 
class_id = 0, name = coa, ap = 96.13%        (TP = 100, FP = 10) 
class_id = 1, name = knight, ap = 100.00%        (TP = 0, FP = 0) 

 for conf_thresh = 0.25, precision = 0.91, recall = 0.93, F1-score = 0.92 
 for conf_thresh = 0.25, TP = 100, FP = 10, FN = 7, average IoU = 74.46 % 

 IoU threshold = 50 %, used Area-Under-Curve for each unique Recall 
 mean average precision (mAP@0.50) = 0.980644, or 98.06 %
beasnap commented 2 years ago

Hello,

I have the same problem. It definately has something to with the detections_count as these vary drasticly in my runs and the Precision does too. I don't know why they are different and what exactly they mean. Is there a way to set these values? I could not find any infos regarding detection_coint, unique_truth_count or ranks.

First time using map:

detections_count = 2018, unique_truth_count = 107  
class_id = 0, name = coa, ap = 62.99%      (TP = 60, FP = 29) 
class_id = 1, name = knight, ap = 100.00%      (TP = 1, FP = 0) 

 for conf_thresh = 0.25, precision = 0.68, recall = 0.57, F1-score = 0.62 
 for conf_thresh = 0.25, TP = 61, FP = 29, FN = 46, average IoU = 49.14 % 

 IoU threshold = 50 %, used Area-Under-Curve for each unique Recall 
 mean average precision (mAP@0.50) = 0.814938, or 81.49 %

Second time using Map

detections_count = 216, unique_truth_count = 107  
 rank = 0 of ranks = 216 
 rank = 100 of ranks = 216 
 rank = 200 of ranks = 216 
class_id = 0, name = coa, ap = 96.13%      (TP = 100, FP = 10) 
class_id = 1, name = knight, ap = 100.00%      (TP = 0, FP = 0) 

 for conf_thresh = 0.25, precision = 0.91, recall = 0.93, F1-score = 0.92 
 for conf_thresh = 0.25, TP = 100, FP = 10, FN = 7, average IoU = 74.46 % 

 IoU threshold = 50 %, used Area-Under-Curve for each unique Recall 
 mean average precision (mAP@0.50) = 0.980644, or 98.06 %

Yes, Hello @DieBirke I found the answer to my problem. Case 1 and Case 2 were completely different. because case 1 is last.weights and case 2 is best.weights. So your problem is also predicted.

Case 1 : If the best.wights file changes while training is in progress Case 2 : If the last.wights file changes while training is in progress Case 3 : Same problem with me. ( Compare last.weights and best.weights )

If not, please describe your situation in detail.

See you!

DieBirke commented 2 years ago

@beasnap

Thanks for your answer. I found my problem. I was looping through directories where my weights are saved but the loop didn't go through the directotires alphabetically. So I just confused different models. Thanks anyway.

aic2022 commented 2 years ago

i have the same problem but with 0: detections_count = 0, unique_truth_count = 0
class_id = 0, name = hand, ap = 0.00% (TP = 0, FP = 0) class_id = 1, name = ball , ap = 0.00% (TP = 0, FP = 0) class_id = 2, name = foot , ap = 0.00% (TP = 0, FP = 0) class_id = 3, name = ground , ap = 0.00% (TP = 0, FP = 0)

for conf_thresh = 0.25, precision = nan, recall = nan, F1-score = nan for conf_thresh = 0.25, TP = 0, FP = 0, FN = 0, average IoU = 0.00 %

IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.000000, or 0.00 % Total Detection Time: 0 Seconds