Open gachiemchiep opened 10 months ago
I also noticed the same for RTMPose.
I had a similar issue for RTMDet instance segmentation. When checking the profiler I realized that the postprocess step was taking 90% of the inference time. That was caused by the huge images I was using as input (something like 2400x1000), and the number of best masks that the model kept.
The size of input image matters because the SDK will resize the results to the original input image, so if you have 100 masks of 640x640 it will resize all of them to the original size and this takes a lot. Consequently, if you reduce the number of masks to resize, the cost will be lower.
The best solution for me would be to have the option to avoid this postprocess step, treating the raw output model as I want and optimizing the process for each use case. However, I have not been able to set up a MMDeploy configuration to do so, so I came with a workaround.
To deal with image resize I previously resize my input image before actually inputing to the model, this way it will output masks of same resized size and it won't take a lot of time resizing masks.
To deal with many masks problem, I lowered to a 10% default value nms_pre and max_per_img, which I found to work really well on my use case, but you just have to evaluate for your specific scenario. Also, I have increased score_thr to 0.5 .
@gnscc
Well it is very odd. Because this C++ inference speed is even slower than python without converting into tensorRT. I guess some python magics does put those post-processing in parallel and it run very fast.
I tried your advice and edit some values inside pipeline.json. But it is still very slow comparing to python inference (without converting to tensorRt)
# converted model's pipeline.json
{
"type": "Task",
"module": "mmdet",
"name": "postprocess",
"component": "ResizeInstanceMask",
"params": {
"mask_thr_binary": 0.5,
"max_per_img": 10,
"min_bbox_size": 0,
"nms": {
"iou_threshold": 0.6,
"type": "nms"
},
"nms_pre": 100,
"score_thr": 0.05,
"is_resize_mask": true
},
"output": [
"post_output"
],
"input": [
"prep_output",
"infer_output"
]
}
Checklist
Describe the bug
I converted rtmdet-ins_s model into tensorrt, then I want to use the C++ inference for ideal speed. But unfortunately, C++ demo run very slow.
I used the tools/profiler.py to check and the result is like this
I tried to compiling mmdeploy and linked against built library, but it didn't help. Is there any method I can use to improve the inference speed of C++ SDK with TensorRT ?
Thanks
Reproduction
No, I didn't
Environment
Error traceback
No response