[Bug] Onnx of ssd300, ssd512 and ssdlite_mobilenetv2_scratch_600e_coco is not giving any bounding boxes

[deepakkupanda]

Checklist

• [X] I have searched related issues but cannot get the expected help.
• [X] 2. I have read the FAQ documentation but cannot get the expected help.
• [X] 3. The bug has not been fixed in the latest version.

Describe the bug

ssd300 and ssd512 do not output any bounding box. I have converted the checkpoint to onnx using this config file "mmdeploy/configs/mmdet/detection/detection_onnxruntime_dynamic.py". The onnx model is successful but when testing the onnx model it does not output any bounding box. I also tried converting the checkpoint using the mmdetection onnx converstion script as well. However, I did not get any result.

Reproduction

python mmdeploy/tools/deploy.py mmdeploy/configs/mmdet/detection/detection_onnxruntime_dynamic.py mmdetection/configs/ssd/ssd512_coco.py mmdetection/checkpoints/ssd512_coco_20210803_022849-0a47a1ca.pth mmdetection/demo/demo.jpg --work-dir mmdeploy_model/ssd512_coco_fp16_dynamic --device cuda --dump-info

Environment

I have used the docker image provided in the github.

Error traceback

No error in the onnx conversion.

[deepakkupanda]

No one has been assigned till now. Please investigate the issue. Thanks

[irexyc]

Sorry for late reply, I can't reproduce your problem with docker image.

When convert the model, there will be two visualized images. Does it looks normal?

# docker run -it --rm --gpus all -v ~/home:/home --network host mmdeploy:cuda
# under workspace
export LD_LIBRARY_PATH=/root/workspace/onnxruntime-linux-x64-1.8.1/lib:$LD_LIBRARY_PATH
python mmdeploy/tools/deploy.py mmdeploy/configs/mmdet/detection/detection_onnxruntime_dynamic.py mmdetection/configs/ssd/ssd512_coco.py https://download.openmmlab.com/mmdetection/v2.0/ssd/ssd512_coco/ssd512_coco_20210803_022849-0a47a1ca.pth mmdetection/demo/demo.jpg --work-dir mmdeploy_model/ssd512_coco_fp16_dynamic --device cuda --dump-info

# then in /root/workspace/mmdeploy_model/ssd512_coco_fp16_dynamic
# there will be two visualized image.
# below is mine.

[deepakkupanda]

The conversion worked for me. I am using my own python script to process the bounding box. The same python file gives output for DETR but does not work for SSD. Please let me know if the bounding box returned by DETR and SSD512 are different? I mean, is bounding box are returning x,y,x,y format or something else. Please let me know.

[deepakkupanda]

One more thing is around the inference time. Usually ssd takes 30-40ms but here converted onnx is taking 330ms. Any reason for this?

[deepakkupanda]

The inference time quoted is in cpu. The pytorch version takes 30-40 ms but the onnx version takes ten times more. Why is it so?

[irexyc]

The return format of dets for detection is same (left, top, right, bottom, score)

When you convert a model, there will be two visualized image, one is inferenced by pytorch and the other is inferenced by backend model. If the result of backend model looks reasonable, the model is likely to be fine. You said the conversion worked for you, did you mean that the visualized result of onnxruntime looked reasonable?

I test ssd512_coco.py on my pc and the result looks fine. The inference time is about 26ms (average for 100 inference times). You said you used same script for DETR and SSD, while according to https://github.com/open-mmlab/mmdetection/blob/master/configs/detr/detr_r50_8x2_150e_coco.py#L72, the std for these two model are not same, the empty ssd output may be caused by wrong std in your scripts?

Below is my test script and visualized result.

import onnxruntime as ort
import cv2
import numpy as np
import time

model = '/root/workspace/mmdeploy_model/ssd512_coco_fp16_dynamic/end2end.onnx'
imgpath = '/root/workspace/mmdetection/demo/demo.jpg'

sess= ort.InferenceSession(model)
mean = np.array([123.675, 116.28, 103.53]).astype(np.float32)
std = np.array([1, 1, 1]).astype(np.float32)

img = cv2.imread(imgpath)
img = cv2.resize(img, (512, 512)).astype(np.float32)
img = img[:, :, ::-1]
img = (img - mean) / std
img = np.transpose(img, (2, 0, 1))
img = img[None]

input_feed = {'input': img}

thres = 0.2
dets, labels = sess.run(['dets', 'labels'], input_feed=input_feed)

img = cv2.imread(imgpath)
for i in range(dets.shape[1]):
    l, t, r, b, s = dets[0, i]
    if s < thres:
        break
    scaleh = 512. / img.shape[0]
    scalew = 512. / img.shape[1]
    l = l / scalew
    r = r / scalew
    t = t / scaleh
    b = b / scaleh
    img = cv2.rectangle(img, (int(l), int(t)), (int(r), int(b)), (0,0,255))

cv2.imwrite('img.jpg', img)

# test inference time
def test_inference_time():
    import time
    st = time.perf_counter()
    loop = 100
    for i in range(loop):
        dets, labels = sess.run(['dets', 'labels'], input_feed=input_feed)

    ed = time.perf_counter()
    print((ed - st) / loop)

test_inference_time()
# 0.026346996149995902