search

ceccocats / tkDNN

Deep neural network library and toolkit to do high performace inference on NVIDIA jetson platforms
GNU General Public License v2.0
718 stars 208 forks source link

yolo3 INT8 conversion drops mAP incredibly #263

Open MohamedElsaeidy opened 2 years ago

MohamedElsaeidy commented 2 years ago

i changed conf_treshhold from 0.3 to 0.001 and tried calibration with 4951 images and still mAP is very bad compared to int8 of yolo4 , the mAP just increased for all models because of conf_tresh except that nothing changed and still bad for yolo3 int8. any suggestion or explanation for what causing that would be greatful. thanks note : all other conversions for fp32 16 int8 for yolo3 and yolo4 are good and very close to published results 

Classes: 80 mAP 0.5:   0.27588
Classes: 80 mAP 0.55:   0.249992
Classes: 80 mAP 0.6:    0.216834
Classes: 80 mAP 0.65:   0.17984
Classes: 80 mAP 0.7:    0.137224
Classes: 80 mAP 0.75:   0.0899831
Classes: 80 mAP 0.8:    0.048967
Classes: 80 mAP 0.85:   0.0190462
Classes: 80 mAP 0.9:    0.00424098
Classes: 80 mAP 0.95:   0.000314722
mAP 0.5:0.95 = 0.122232
avg precision: 0.119544 avg recall: 0.523164    avg f1 score:0.194617
mive93 commented 2 years ago

Hi @MohamedElsaeidy, It is normal a drop of performance when going from float (even half precision) to int8. I do agree that the jump is very big. Here I can attach a chart with several comparison we did for a journal paper (still under review), in which you can find a lor of information about different platforms and network for all the data types. Hope it helps.

image