search

open-mmlab / mmpose

OpenMMLab Pose Estimation Toolbox and Benchmark.
https://mmpose.readthedocs.io/en/latest/
Apache License 2.0
5.55k stars 1.21k forks source link

如何使用rtmo的SDK版onnx模型进行推理 #2915

Open lzk020813 opened 8 months ago

lzk020813 commented 8 months ago

Prerequisite

Environment

Python' : '3.8.18 CUDA available' : True PyTorch' : '1.10.0

Reproduces the problem - code sample

我想使用rtmo的SDK版onnx模型进行推理，用mmdeploy中的pose_detection.py进行测试，代码如下

import argparse
import cv2
import numpy as np
from mmdeploy_runtime import PoseDetector

def parse_args():
    parser = argparse.ArgumentParser(
        description='show how to use sdk python api')
    parser.add_argument('device_name', help='name of device, cuda or cpu')
    parser.add_argument(
        'model_path',
        help='path of mmdeploy SDK model dumped by model converter')
    parser.add_argument('image_path', help='path of an image')
    parser.add_argument(
        '--bbox',
        default=None,
        nargs='+',
        type=int,
        help='bounding box of an object in format (x, y, w, h)')
    args = parser.parse_args()
    return args

def main():
    args = parse_args()

    img = cv2.imread(args.image_path)

    detector = PoseDetector(
        model_path=args.model_path, device_name=args.device_name, device_id=0)

    if args.bbox is None:
        result = detector(img)
    else:
        # converter (x, y, w, h) -> (left, top, right, bottom)
        print(args.bbox)
        bbox = np.array(args.bbox, dtype=int)
        bbox[2:] += bbox[:2]
        result = detector(img, bbox)
    print(result)

    _, point_num, _ = result.shape
    points = result[:, :, :2].reshape(point_num, 2)
    for [x, y] in points.astype(int):
        cv2.circle(img, (x, y), 1, (0, 255, 0), 2)

    cv2.imwrite('output_pose.png', img)

if __name__ == '__main__':
    main()

Reproduces the problem - command or script

运行命令 python pose_detection.py cpu d:/rtmpose-ort/rtmo d:/1.jpg 进行测试，其中d:/rtmpose-ort/rtmo为sdk目录，但好像无法实现

Reproduces the problem - error message

出现以下信息 

loading mmdeploy_trt_net.dll ...
failed to load library mmdeploy_trt_net.dll
loading mmdeploy_ort_net.dll ...
[2024-01-06 15:41:32.149] [mmdeploy] [info] [model.cpp:35] [DirectoryModel] Load model: "d:/rtmpose-ort/rtmo"
[2024-01-06 15:41:32.149] [mmdeploy] [error] [compose.cpp:37] Unable to find Transform creator: BottomupResize. Available transforms: [("CenterCrop", 0), ("Collect", 0), ("Compose", 0), ("DefaultFormatBund
le", 0), ("FormatShape", 0), ("ImageToTensor", 0), ("LetterResize", 0), ("Lift", 0), ("LoadImageFromFile", 0), ("Normalize", 0), ("Pad", 0), ("RescaleToHeight", 0), ("Resize", 0), ("ResizeOCR", 0), ("Short
ScaleAspectJitter", 0), ("TenCrop", 0), ("ThreeCrop", 0), ("TopDownAffine", 0), ("TopDownGetBboxCenterScale", 0)]
[2024-01-06 15:41:32.149] [mmdeploy] [error] [task.cpp:99] error parsing config: {
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "img"
  ],
  "module": "Transform",
  "name": "Preprocess",
  "output": [
    "prep_output"
  ],
  "transforms": [
    {
      "type": "LoadImageFromFile"
    },
    {
      "input_size": [
        640,
        640
      ],
      "pad_val": [
        114,
        114,
        114
      ],
      "type": "BottomupResize"
    },
    {
      "mean": [
        0,
        0,
        0
      ],
      "std": [
        1,
        1,
        1
      ],
      "to_rgb": false,
      "type": "Normalize"
    },
    {
      "keys": [
        "img"
      ],
      "type": "ImageToTensor"
    },
    {
      "keys": [
        "img"
      ],
      "meta_keys": [
        "img_shape",
        "pad_shape",
        "ori_shape",
        "img_norm_cfg",
        "scale_factor",
        "bbox_score",
        "center",
        "scale"
      ],
      "type": "Collect"
    }
  ],
  "type": "Task"
}
[2024-01-06 15:41:32.474] [mmdeploy] [error] [common.h:50] Could not found entry 'UNKNOWN' in mmpose. Available components: [("DeepposeRegressionHeadDecode", 0), ("SimCCLabelDecode", 0), ("TopdownHeatmapBa
seHeadDecode", 0), ("TopdownHeatmapMSMUHeadDecode", 0), ("TopdownHeatmapMultiStageHeadDecode", 0), ("TopdownHeatmapSimpleHeadDecode", 0), ("ViPNASHeatmapSimpleHeadDecode", 0)]
[2024-01-06 15:41:32.475] [mmdeploy] [error] [task.cpp:99] error parsing config: {
  "component": "UNKNOWN",
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "prep_output",
    "infer_output"
  ],
  "module": "mmpose",
  "name": "postprocess",
  "output": [
    "post_output"
  ],
  "params": {
    "flip_test": false,
    "input_size": [
      640,
      640
    ],
    "nms_thr": 0.65,
    "score_thr": 0.1,
    "type": "YOLOXPoseAnnotationProcessor"
  },
  "type": "Task"
}

Additional information

之前没有用sdk模型处理过one-stage的算法，请问我该怎样才能用rtmo的SDK版onnx模型进行推理呢，十分感谢

Ben-Louis commented 7 months ago

可以尝试下 rtmlib

danigarciaoca commented 7 months ago

Same problem on my side, when converting RTMO to TRT and trying to perform inference with MMDeploy

sxj731533730 commented 7 months ago

ubuntu@ubuntu:~/mmdeploy$ python3 tools/deploy.py configs/mmpose/pose-detection_rtmo_tensorrt-fp16_dynamic-640x640.py /home/ubuntu/mmpose/configs/body_2d_keypoint/rtmo/body7/rtmo-t_8xb32-600e_body7-416x416.py https://download.openmmlab.com/mmpose/v1/projects/rtmo/rtmo-t_8xb32-600e_body7-416x416-f48f75cb_20231219.pth /home/ubuntu/mmpose/tests/data/coco/000000000785.jpg --work-dir onnx_dir --dump-info --show --device cuda:0

face the problem

RuntimeError: Unsupported: ONNX export of transpose for tensor of unknown rank. 01/30 08:46:38 - mmengine - ERROR - /home/ubuntu/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - mmdeploy.apis.pytorch2onnx.torch2onnx with Call id: 0 failed. exit.

########################################################

ubuntu@ubuntu:~/mmdeploy$ python3 tools/deploy.py configs/mmpose/pose-detection_rtmo_onnxruntime_dynamic.py /home/ubuntu/mmpose/configs/body_2d_keypoint/rtmo/body7/rtmo-t_8xb32-600e_body7-416x416.py https://download.openmmlab.com/mmpose/v1/projects/rtmo/rtmo-t_8xb32-600e_body7-416x416-f48f75cb_20231219.pth /home/ubuntu/mmpose/tests/data/coco/000000000785.jpg --work-dir onnx_dir --dump-info --show --device cuda:0

face the problem

RuntimeError: Exporting the operator linalg_norm to ONNX opset version 11 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub. 01/30 08:35:59 - mmengine - ERROR - /home/ubuntu/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - mmdeploy.apis.pytorch2onnx.torch2onnx with Call id: 0 failed. exit.

luuuyi commented 5 months ago

ubuntu@ubuntu:~/mmdeploy$ python3 tools/deploy.py configs/mmpose/pose-detection_rtmo_tensorrt-fp16_dynamic-640x640.py /home/ubuntu/mmpose/configs/body_2d_keypoint/rtmo/body7/rtmo-t_8xb32-600e_body7-416x416.py https://download.openmmlab.com/mmpose/v1/projects/rtmo/rtmo-t_8xb32-600e_body7-416x416-f48f75cb_20231219.pth /home/ubuntu/mmpose/tests/data/coco/000000000785.jpg --work-dir onnx_dir --dump-info --show --device cuda:0

face the problem

RuntimeError: Unsupported: ONNX export of transpose for tensor of unknown rank. 01/30 08:46:38 - mmengine - ERROR - /home/ubuntu/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - mmdeploy.apis.pytorch2onnx.torch2onnx with Call id: 0 failed. exit.

########################################################

ubuntu@ubuntu:~/mmdeploy$ python3 tools/deploy.py configs/mmpose/pose-detection_rtmo_onnxruntime_dynamic.py /home/ubuntu/mmpose/configs/body_2d_keypoint/rtmo/body7/rtmo-t_8xb32-600e_body7-416x416.py https://download.openmmlab.com/mmpose/v1/projects/rtmo/rtmo-t_8xb32-600e_body7-416x416-f48f75cb_20231219.pth /home/ubuntu/mmpose/tests/data/coco/000000000785.jpg --work-dir onnx_dir --dump-info --show --device cuda:0

face the problem

RuntimeError: Exporting the operator linalg_norm to ONNX opset version 11 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub. 01/30 08:35:59 - mmengine - ERROR - /home/ubuntu/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - mmdeploy.apis.pytorch2onnx.torch2onnx with Call id: 0 failed. exit.

also face this problem

luuuyi commented 5 months ago

可以尝试下 rtmlib

能问下你的pytorch是什么版本的么，转rtmo的op_set是多少

Ben-Louis commented 5 months ago

@luuuyi 截屏2024-03-21 11 31 49

zephyrzhu1998 commented 4 months ago

@lzk020813 你好，我也遇到了这个问题，请问你后来解决了吗？