mAP of deployed yolov5x decreased

[chiyukunpeng]

hi, I infer surveillance videos with yolov5x according to https://github.com/wang-xinyu/tensorrtx/tree/master/yolov5. mAP@.50 of that is 94.8%.

After deploying yolov5x with triton20.08, I found that mAP@.50 decreased.

Do you know the reasons for that? Thanks!

triton client output

Frame 213: 90 raw boxes, 4 objects
CAR:     0.95
CAR:     0.92
CAR:     0.85
CHEMICAL_VEHICLE:     0.74
time:    28.7ms
Frame 214: 91 raw boxes, 4 objects
CAR:     0.94
CAR:     0.93
CAR:     0.86
CHEMICAL_VEHICLE:     0.72
time:    28.5ms
Frame 215: 104 raw boxes, 4 objects
CAR:     0.94
CAR:     0.93
CAR:     0.85
TRUCK:     0.56
time:    28.0ms
Frame 216: 91 raw boxes, 4 objects
CAR:     0.95
CAR:     0.94
CAR:     0.86
TRUCK:     0.74
time:    28.1ms
Frame 217: 90 raw boxes, 4 objects
CAR:     0.95
CAR:     0.94
CAR:     0.86
TRUCK:     0.74
time:    27.7ms
Frame 218: 107 raw boxes, 4 objects
CAR:     0.95
CAR:     0.95
CAR:     0.88
TRUCK:     0.76
time:    27.9ms

postprocess func

def postprocess(buffer, image_width, image_height, conf_threshold=0.8, nms_threshold=0.5):
    detected_objects = []
    img_scale = [image_width / INPUT_WIDTH, image_height / INPUT_HEIGHT, image_width / INPUT_WIDTH, image_height / INPUT_HEIGHT]
    num_bboxes = int(buffer[0, 0, 0, 0])

    if num_bboxes:
        bboxes = buffer[0, 1 : (num_bboxes * 6 + 1), 0, 0].reshape(-1, 6)
        labels = set(bboxes[:, 5].astype(int))

        for label in labels:
            selected_bboxes = bboxes[np.where((bboxes[:, 5] == label) & (bboxes[:, 4] >= conf_threshold))]
            selected_bboxes_keep = selected_bboxes[nms(selected_bboxes[:, :4], selected_bboxes[:, 4], nms_threshold)]
            for idx in range(selected_bboxes_keep.shape[0]):
                box_xy = selected_bboxes_keep[idx, :2]
                box_wh = selected_bboxes_keep[idx, 2:4]
                score = selected_bboxes_keep[idx, 4]

                box_x1y1 = box_xy - (box_wh / 2)
                box_x2y2 = np.minimum(box_xy + (box_wh / 2), [INPUT_WIDTH, INPUT_HEIGHT])
                box = np.concatenate([box_x1y1, box_x2y2])
                box *= img_scale

                if box[0] == box[2]:
                    continue
                if box[1] == box[3]:
                    continue
                detected_objects.append(BoundingBox(label, score, box[0], box[2], box[1], box[3], image_height, image_width))
    return detected_objects

[philipp-schmidt]

By how much is your precision decreased? How do you measure mAP in both cases?

[chiyukunpeng]

@philipp-schmidt hi, I unframed two output videos. I found that the video after deployment has false detections but there are no false detections before deployment.

[philipp-schmidt]

Hi, Yes, I have experienced similar problems and reported them in 2 issues in the triton repo. Can you share more details so I can add them to the issue?

E.g. are you using multiple clients at the same time? How fast are you retrieving results? How is the error looking? Only missing detections or wrong detections as well?

[philipp-schmidt]

I believe the following issues are related to this problem:

https://github.com/triton-inference-server/server/issues/2231 https://github.com/triton-inference-server/server/issues/2135

And probably also this one: https://github.com/triton-inference-server/server/issues/2322

I would ask you to give me as much info that you can give away so I can get a proper bug report going. How many GPUs do you use?

[philipp-schmidt]

Please also try without FP16 optimization:

https://github.com/isarsoft/yolov4-triton-tensorrt/blob/4edbe849f111324d58eda5f14fe1e4e9b098d4cd/networks/yolov4.h#L10

[chiyukunpeng]

@philipp-schmidt Hi, it is my pleasure to share more details.

• Are you using multiple clients at the same time? No.

• How fast are you retrieving results? It infers videos with CPU. I want to infer it with GPU. Can you help me? Thanks!

  Frame 248: 73 raw boxes, 3 objects
  car:     0.93
  car:     0.92
  car:     0.92
  time:    27.8ms

• How is the error looking? Only missing detections or wrong detections as well? My errors include missing detections and wrong detections.

• How many GPUs do you use? one 2080 GPU.

• Please also try without FP16 optimization Missing detections and wrong detections decreases, but they still exist.

[philipp-schmidt]

How do you measure your mAP and can you share an example of an image with correct detections before deployment and wrong detections after? How do you validate without triton?

[chiyukunpeng]

wrong detections

before deployment after deployment

misssing detections

before deployment after deployment

[philipp-schmidt]

I don't see what's wrong here. Truck and Chemical are swapped because your class indices have to be changed in the python client code, search for the COCOLabels in the code.

The other image might simply be due to different detection thresholds. Make sure they are identical and compare again.

[chiyukunpeng]

Thanks!