mAPs of some networks are much lower than expected.

[chjej202]

Hello,

I tested multiple pre-trained weights provided by tkDNN with "map_demo." However, mAP of some networks are lower than expected. (mobilenetv2ssd512, yolo3tiny, dla34_cnet, resnet101_cnet)

mAP of yolo3tiny is too small. (mAP 0.5 : 0.12) mAP of mobilenetv2ssd512 is too small even though input image is larger (512x512) than yolo (416x416) variants. mAP of dla34_cnet and resnet101_cnet is not competitive to yolov3 which is an old network compared to cnet.

Do you have mAP results of all the networks which can run with "map_demo"?

I ran the program on Nvidia Jetson AGX Xavier with Jetpack 4.3 I used coco 2017 valid set with 4952 images to check the accuracy of each network. All models are built with FP16 mode.

The below results shows the mAP of each network.

cspresnext50-panet-spp Classes: 80 mAP 0.5: 0.638923 Classes: 80 mAP 0.55: 0.620606 Classes: 80 mAP 0.6: 0.598838 Classes: 80 mAP 0.65: 0.567956 Classes: 80 mAP 0.7: 0.524836 Classes: 80 mAP 0.75: 0.47366 Classes: 80 mAP 0.8: 0.393327 Classes: 80 mAP 0.85: 0.270965 Classes: 80 mAP 0.9: 0.128609 Classes: 80 mAP 0.95: 0.0145363 mAP 0.5:0.95 = 0.423226 avg precision: 0.80101 avg recall: 0.669992 avg f1 score:0.729667

yolo4tiny Classes: 80 mAP 0.5: 0.276675 Classes: 80 mAP 0.55: 0.262956 Classes: 80 mAP 0.6: 0.245394 Classes: 80 mAP 0.65: 0.220419 Classes: 80 mAP 0.7: 0.188917 Classes: 80 mAP 0.75: 0.145544 Classes: 80 mAP 0.8: 0.0963829 Classes: 80 mAP 0.85: 0.0461462 Classes: 80 mAP 0.9: 0.0122035 Classes: 80 mAP 0.95: 0.000870022 mAP 0.5:0.95 = 0.149551 avg precision: 0.661997 avg recall: 0.307922 avg f1 score:0.420331

yolov4 Classes: 80 mAP 0.5: 0.59428 Classes: 80 mAP 0.55: 0.576454 Classes: 80 mAP 0.6: 0.556307 Classes: 80 mAP 0.65: 0.524986 Classes: 80 mAP 0.7: 0.487346 Classes: 80 mAP 0.75: 0.434977 Classes: 80 mAP 0.8: 0.364772 Classes: 80 mAP 0.85: 0.262967 Classes: 80 mAP 0.9: 0.139866 Classes: 80 mAP 0.95: 0.0178965 mAP 0.5:0.95 = 0.395985 avg precision: 0.757215 avg recall: 0.633702 avg f1 score:0.689974

yolo3tiny Classes: 80 mAP 0.5: 0.120749 Classes: 80 mAP 0.55: 0.115145 Classes: 80 mAP 0.6: 0.108731 Classes: 80 mAP 0.65: 0.0975029 Classes: 80 mAP 0.7: 0.0819545 Classes: 80 mAP 0.75: 0.0617638 Classes: 80 mAP 0.8: 0.0400224 Classes: 80 mAP 0.85: 0.0181904 Classes: 80 mAP 0.9: 0.00465814 Classes: 80 mAP 0.95: 0.0005046 mAP 0.5:0.95 = 0.0649222 avg precision: 0.438526 avg recall: 0.168005 avg f1 score:0.242937

yolo3 Classes: 80 mAP 0.5: 0.53915 Classes: 80 mAP 0.55: 0.518971 Classes: 80 mAP 0.6: 0.49056 Classes: 80 mAP 0.65: 0.450069 Classes: 80 mAP 0.7: 0.400232 Classes: 80 mAP 0.75: 0.324907 Classes: 80 mAP 0.8: 0.227801 Classes: 80 mAP 0.85: 0.118608 Classes: 80 mAP 0.9: 0.0299095 Classes: 80 mAP 0.95: 0.00156115 mAP 0.5:0.95 = 0.310177 avg precision: 0.734336 avg recall: 0.573578 avg f1 score:0.644077

mobilentv2ssd512 Classes: 80 mAP 0.5: 0.298511 Classes: 80 mAP 0.55: 0.289307 Classes: 80 mAP 0.6: 0.274653 Classes: 80 mAP 0.65: 0.256511 Classes: 80 mAP 0.7: 0.232621 Classes: 80 mAP 0.75: 0.204093 Classes: 80 mAP 0.8: 0.164697 Classes: 80 mAP 0.85: 0.112635 Classes: 80 mAP 0.9: 0.0504613 Classes: 80 mAP 0.95: 0.00523403 mAP 0.5:0.95 = 0.188872 avg precision: 0.640366 avg recall: 0.334324 avg f1 score:0.439298

resnet101_cnet Classes: 80 mAP 0.5: 0.438912 Classes: 80 mAP 0.55: 0.424819 Classes: 80 mAP 0.6: 0.406554 Classes: 80 mAP 0.65: 0.384212 Classes: 80 mAP 0.7: 0.354354 Classes: 80 mAP 0.75: 0.312488 Classes: 80 mAP 0.8: 0.262612 Classes: 80 mAP 0.85: 0.199722 Classes: 80 mAP 0.9: 0.116854 Classes: 80 mAP 0.95: 0.0265936 mAP 0.5:0.95 = 0.292712 avg precision: 0.606124 avg recall: 0.494776 avg f1 score:0.544819

dla34_cnet Classes: 80 mAP 0.5: 0.463488 Classes: 80 mAP 0.55: 0.449222 Classes: 80 mAP 0.6: 0.42911 Classes: 80 mAP 0.65: 0.408078 Classes: 80 mAP 0.7: 0.379758 Classes: 80 mAP 0.75: 0.341952 Classes: 80 mAP 0.8: 0.293338 Classes: 80 mAP 0.85: 0.226469 Classes: 80 mAP 0.9: 0.138922 Classes: 80 mAP 0.95: 0.032453 mAP 0.5:0.95 = 0.316279 avg precision: 0.569139 avg recall: 0.536718 avg f1 score:0.552453

[mive93]

Hi @chjej202 yes, it is true. The problem is that (I'm sure) you used confidence threshold = 0.3. It changes a lot the results indeed. Try using 0.05 instead and you will get the correct values.

[chjej202]

Hi, I changed a confidence threshold (conf_threshold) in DetectionNN.h from 0.3 to 0.05. It increases mAP of all networks, but I still don't know some networks (dla34_cnet, resnet101_cnet, mobilenetv2ssd, yolov3tiny) work poorly compared to other networks.

Do you have any reference results tested by map_demo to check my results are correct or not?

=========================================================================== Below results are my test results with conf_threshold = 0.05

yolov3tiny (416x416, conf_threshold = 0.05) Classes: 80 mAP 0.5: 0.180512 Classes: 80 mAP 0.55: 0.166291 Classes: 80 mAP 0.6: 0.150633 Classes: 80 mAP 0.65: 0.129194 Classes: 80 mAP 0.7: 0.103001 Classes: 80 mAP 0.75: 0.0742617 Classes: 80 mAP 0.8: 0.0453536 Classes: 80 mAP 0.85: 0.0195794 Classes: 80 mAP 0.9: 0.00479772 Classes: 80 mAP 0.95: 0.000505335 mAP 0.5:0.95 = 0.0874129 avg precision: 0.184815 avg recall: 0.327286 avg f1 score:0.236232

mobilenetv2ssd512 (512x512, conf_threshold = 0.05) Classes: 80 mAP 0.5: 0.376689 Classes: 80 mAP 0.55: 0.359028 Classes: 80 mAP 0.6: 0.334558 Classes: 80 mAP 0.65: 0.304146 Classes: 80 mAP 0.7: 0.267701 Classes: 80 mAP 0.75: 0.228689 Classes: 80 mAP 0.8: 0.180935 Classes: 80 mAP 0.85: 0.120037 Classes: 80 mAP 0.9: 0.0523691 Classes: 80 mAP 0.95: 0.00519432 mAP 0.5:0.95 = 0.222935

resnet101_cnet (512x512, conf_threshold = 0.05) Classes: 80 mAP 0.5: 0.502354 Classes: 80 mAP 0.55: 0.481269 Classes: 80 mAP 0.6: 0.454985 Classes: 80 mAP 0.65: 0.425261 Classes: 80 mAP 0.7: 0.387514 Classes: 80 mAP 0.75: 0.337049 Classes: 80 mAP 0.8: 0.279151 Classes: 80 mAP 0.85: 0.209493 Classes: 80 mAP 0.9: 0.120783 Classes: 80 mAP 0.95: 0.0270623 mAP 0.5:0.95 = 0.322492 avg precision: 0.0872499 avg recall: 0.683761 avg f1 score:0.154753

dla34_cnet (512x512, conf_threshold = 0.05) Classes: 80 mAP 0.5: 0.525197 Classes: 80 mAP 0.55: 0.506484 Classes: 80 mAP 0.6: 0.480806 Classes: 80 mAP 0.65: 0.453351 Classes: 80 mAP 0.7: 0.41684 Classes: 80 mAP 0.75: 0.369797 Classes: 80 mAP 0.8: 0.313306 Classes: 80 mAP 0.85: 0.238509 Classes: 80 mAP 0.9: 0.14341 Classes: 80 mAP 0.95: 0.0331084 mAP 0.5:0.95 = 0.348081 avg precision: 0.0990177 avg recall: 0.740642 avg f1 score:0.174682

yolov4tiny (416x416, conf_threshold = 0.05) Classes: 80 mAP 0.5: 0.371197 Classes: 80 mAP 0.55: 0.345488 Classes: 80 mAP 0.6: 0.314449 Classes: 80 mAP 0.65: 0.274678 Classes: 80 mAP 0.7: 0.223176 Classes: 80 mAP 0.75: 0.164865 Classes: 80 mAP 0.8: 0.10477 Classes: 80 mAP 0.85: 0.0487025 Classes: 80 mAP 0.9: 0.012689 Classes: 80 mAP 0.95: 0.000883334 mAP 0.5:0.95 = 0.18609 avg precision: 0.289453 avg recall: 0.501469 avg f1 score:0.367044

yolov4 (416x416, conf_threshold = 0.05) Classes: 80 mAP 0.5: 0.672525 Classes: 80 mAP 0.55: 0.645826 Classes: 80 mAP 0.6: 0.615812 Classes: 80 mAP 0.65: 0.574007 Classes: 80 mAP 0.7: 0.52595 Classes: 80 mAP 0.75: 0.463363 Classes: 80 mAP 0.8: 0.381599 Classes: 80 mAP 0.85: 0.271517 Classes: 80 mAP 0.9: 0.142526 Classes: 80 mAP 0.95: 0.0179826 mAP 0.5:0.95 = 0.431111 avg precision: 0.426695 avg recall: 0.771391 avg f1 score:0.549458

yolov3 (416x416, conf_threshold = 0.05) Classes: 80 mAP 0.5: 0.623093 Classes: 80 mAP 0.55: 0.592053 Classes: 80 mAP 0.6: 0.550943 Classes: 80 mAP 0.65: 0.499786 Classes: 80 mAP 0.7: 0.436613 Classes: 80 mAP 0.75: 0.34891 Classes: 80 mAP 0.8: 0.238972 Classes: 80 mAP 0.85: 0.122449 Classes: 80 mAP 0.9: 0.0305109 Classes: 80 mAP 0.95: 0.00156678 mAP 0.5:0.95 = 0.34449 avg precision: 0.42965 avg recall: 0.721075 avg f1 score:0.53846

csresnext50-panet-spp (416x416, conf_threshold = 0.05) Classes: 80 mAP 0.5: 0.722289 Classes: 80 mAP 0.55: 0.69365 Classes: 80 mAP 0.6: 0.658735 Classes: 80 mAP 0.65: 0.616528 Classes: 80 mAP 0.7: 0.559819 Classes: 80 mAP 0.75: 0.497859 Classes: 80 mAP 0.8: 0.406925 Classes: 80 mAP 0.85: 0.276912 Classes: 80 mAP 0.9: 0.129772 Classes: 80 mAP 0.95: 0.0145543 mAP 0.5:0.95 = 0.457704 avg precision: 0.495352 avg recall: 0.802046 avg f1 score:0.61245

[mive93]

HI @chjej202,

No idea why. I will check all of them tomorrow and will give you a proper answer

[chjej202]

Thank you. I will wait for your answer.

[mive93]

Hi @chjej202,

sorry for the wait. It actually took me till today to find out and solve the issue. Here you can find what I did : AlexeyAB/darknet#5354 (comment). Moreover, here's the current mAP results of tkDNN https://github.com/ceccocats/tkDNN#map-results.

The results are now coherent to darknet :) Notice that the values of the map differ from the ones in the paper: this is due to the different testset, which is not test-dev, but COCO val 2017 instead.

Also, tkDNN mAP computation is a bit pessimistic wrt the one by CodaLab, but I still don't know why.

[chjej202]

Thank you for your reply. :) My mAP is similar to your results, so I will close the issue.