search

xiaocao-tian / yolov8_tensorrt

pt-->wts-->engine
127 stars 14 forks source link

Lower mAP as compared to other implementation #2

Open xiang-wuu opened 1 year ago

xiang-wuu commented 1 year ago

From this implementation inference over validation dataset as per below table and the similar implementation provided in triple-Mu/YOLOv8-TensorRT using TRT network builder API which provides NMS plugin as a part of TRT network produces significantly better results as compared to the implementation in this repo. in below comparison table i computed mAP of 3 object categories from COCO dataset, there is huge gap in the mAP across two implementations all the parameters are kept same like (iou_threshold=0.45, conf_threshold=0.30, precision=FP16, input shpae=608x608) while producing the below results.

Dataset TensorRT-8.4 (NVIDIA GeForce RTX 3080 12GB) (FP16 Precision) Img: 608 X 608
Yolov8 (xiaocao-tian/yolov8_tensorrt) Yolov8 (triple-Mu/YOLOv8-TensorRT)
Accuracy (mAP) @ IoU=0.75 Accuracy (mAP) @ IoU=0.75
COCO Person - 55.4 42.4 Person - 74 52.33
Vehicle - 50.6 Vehicle - 62.2
Bag Object - 21.2 Bag Object - 21.7

As it can be seen from below table , the image to the left doesn't detect two persons which certainly getting detected by the implementation in the right image, there are many such images where objects are visible clearly but it's not getting detected with this repo's implementation.

Yolov8 (xiaocao-tian/yolov8_tensorrt) Yolov8 (triple-Mu/YOLOv8-TensorRT)

Observations: There are multiple different scenarios in images some of which are very easy to detect but it's not get detected where as the other implementation detects some of the most difficult appearing objects in the image.

Possible Cause: As i tried changing the inference shape while serializing the engine so input shape at resolution 608x608 produces better results as compared to 640x640, from my observations pre-processing function might be the main culprit to this issue, however i tried implementing the pre-processing function as in here while inference, but the results are still identical i am not sure if am missing anything else, any help on this would be greatly appreciated.

guanshanjushi commented 1 year ago

我看这个表格发现是不是有些层的实现和pytorch的实现不一致，导致精度不对齐？

guanshanjushi commented 1 year ago

或者该版本的实现不是最新的u版yolov8？

xiang-wuu commented 1 year ago

或者该版本的实现不是最新的u版yolov8？

yolov8版本是最新的

xiaocao-tian commented 1 year ago

方便提供一下这张图片吗？

xiang-wuu commented 1 year ago

方便提供一下这张图片吗？

谢谢，我过一会儿就发帖!