search

open-mmlab / mmdeploy

OpenMMLab Model Deployment Framework
https://mmdeploy.readthedocs.io/en/latest/
Apache License 2.0
2.71k stars 619 forks source link

[Bug] MMdeploy does not work with cuda. #2069

Closed CerPhd closed 1 year ago

CerPhd commented 1 year ago

Checklist

Describe the bug

Hi everyone I have installed all the libraries needed to inference a video with mmdeploy. If I use cpu for extrapolating human pose from a video (CODE1) everything is fine. When I use cuda (CODE2) , I have Error1. I thought maybe cuda was not installed properly, but when I run the sample code CODE3 , it is working! Thank you for your attention.

Reproduction

CODE1 python pose_tracker.py cpu rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ cam2.mp4

CODE2 python pose_tracker.py cuda rtmpose-ort/rtmdet-nano/ rtmpose-ort/rtmpose-m/ cam2.mp4

CODE3 (sample code from tutorial, from that I suppose that cuda is properly working) python tools/deploy.py configs/mmpose/pose-detection_simcc_tensorrt_dynamic-256x192.py C:\Users\lucac\src\mmpose\configs\wholebody\2d_kpt_sview_rgb_img\topdown_heatmap\coco-wholebody\tcformer_coco_wholebody_256x192.py https://download.openmmlab.com/mmpose/top_down/tcformer/tcformer_coco-wholebody_256x192-a0720efa_20220627.pth C:\Users\lucac\Pictures\ppp.jpg --work-dir rtmpose-trt/rtmpose-m --device cuda:0 --show --dump-info

Environment

This is my env (using "check_env.py")

**Output check env**
"05/10 17:28:18 - mmengine - INFO - sys.platform: win32
05/10 17:28:18 - mmengine - INFO - Python: 3.8.16 | packaged by conda-forge | (default, Feb  1 2023, 15:53:35) [MSC v.1929 64 bit (AMD64)]
05/10 17:28:18 - mmengine - INFO - CUDA available: True
05/10 17:28:18 - mmengine - INFO - numpy_random_seed: 2147483648
05/10 17:28:18 - mmengine - INFO - GPU 0: NVIDIA GeForce RTX 3080 Laptop GPU
05/10 17:28:18 - mmengine - INFO - CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7
05/10 17:28:18 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.7, V11.7.64
05/10 17:28:18 - mmengine - INFO - MSVC: Microsoft (R) C/C++ Optimizing Compiler versione 19.33.31630 per x64
05/10 17:28:18 - mmengine - INFO - GCC: n/a
05/10 17:28:18 - mmengine - INFO - PyTorch: 1.12.1+cu113
05/10 17:28:18 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - C++ Version: 199711
  - MSVC 192829337
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 2019
  - LAPACK is enabled (usually provided by MKL)
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.3.2  (built against CUDA 11.5)
  - Magma 2.5.4
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=C:/actions-runner/_work/pytorch/pytorch/builder/windows/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/actions-runner/_work/pytorch/pytorch/builder/windows/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

05/10 17:28:18 - mmengine - INFO - TorchVision: 0.13.1+cu113
05/10 17:28:18 - mmengine - INFO - OpenCV: 4.7.0
05/10 17:28:18 - mmengine - INFO - MMEngine: 0.7.3
05/10 17:28:18 - mmengine - INFO - MMCV: 2.0.0
05/10 17:28:18 - mmengine - INFO - MMCV Compiler: MSVC 192829924
05/10 17:28:18 - mmengine - INFO - MMCV CUDA Compiler: 11.3
05/10 17:28:18 - mmengine - INFO - MMDeploy: 1.0.0+162f4cb
05/10 17:28:18 - mmengine - INFO -

05/10 17:28:18 - mmengine - INFO - **********Backend information**********
05/10 17:28:18 - mmengine - INFO - tensorrt:    8.2.3.0
05/10 17:28:18 - mmengine - INFO - tensorrt custom ops: NotAvailable
05/10 17:28:18 - mmengine - INFO - ONNXRuntime: 1.8.1
05/10 17:28:18 - mmengine - INFO - ONNXRuntime-gpu:     1.14.1
05/10 17:28:18 - mmengine - INFO - ONNXRuntime custom ops:      NotAvailable
05/10 17:28:18 - mmengine - INFO - pplnn:       None
05/10 17:28:18 - mmengine - INFO - ncnn:        None
05/10 17:28:18 - mmengine - INFO - snpe:        None
05/10 17:28:18 - mmengine - INFO - openvino:    None
05/10 17:28:18 - mmengine - INFO - torchscript: 1.12.1+cu113
05/10 17:28:18 - mmengine - INFO - torchscript custom ops:      NotAvailable
05/10 17:28:18 - mmengine - INFO - rknn-toolkit:        None
05/10 17:28:18 - mmengine - INFO - rknn-toolkit2:       None
05/10 17:28:18 - mmengine - INFO - ascend:      None
05/10 17:28:18 - mmengine - INFO - coreml:      None
05/10 17:28:18 - mmengine - INFO - tvm: None
05/10 17:28:18 - mmengine - INFO - vacc:        None
05/10 17:28:18 - mmengine - INFO -

05/10 17:28:18 - mmengine - INFO - **********Codebase information**********
05/10 17:28:18 - mmengine - INFO - mmdet:       3.0.0
05/10 17:28:18 - mmengine - INFO - mmseg:       None
05/10 17:28:18 - mmengine - INFO - mmpretrain:  None
05/10 17:28:18 - mmengine - INFO - mmocr:       None
05/10 17:28:18 - mmengine - INFO - mmedit:      None
05/10 17:28:18 - mmengine - INFO - mmdet3d:     None
05/10 17:28:18 - mmengine - INFO - mmpose:      1.0.0
05/10 17:28:18 - mmengine - INFO - mmrotate:    None
05/10 17:28:18 - mmengine - INFO - mmaction:    None
05/10 17:28:18 - mmengine - INFO - mmrazor:     None"

**End Output check env**

CONDA Env (cmd conda list)

addict                    2.4.0                    pypi_0    pypi
aenum                     3.1.12                   pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
attrs                     23.1.0                   pypi_0    pypi
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2023.5.7             h56e8100_0    conda-forge
certifi                   2023.5.7                 pypi_0    pypi
charset-normalizer        3.1.0                    pypi_0    pypi
chumpy                    0.70                     pypi_0    pypi
click                     8.1.3                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
coloredlogs               15.0.1                   pypi_0    pypi
contourpy                 1.0.7                    pypi_0    pypi
coverage                  7.2.5                    pypi_0    pypi
cudatoolkit               11.3.1               h59b6b97_2
cycler                    0.11.0                   pypi_0    pypi
cython                    0.29.34                  pypi_0    pypi
dill                      0.3.6                    pypi_0    pypi
exceptiongroup            1.1.1                    pypi_0    pypi
flake8                    6.0.0                    pypi_0    pypi
flatbuffers               23.5.8                   pypi_0    pypi
fonttools                 4.39.3                   pypi_0    pypi
grpcio                    1.54.0                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
idna                      3.4                      pypi_0    pypi
importlib-metadata        6.6.0                    pypi_0    pypi
importlib-resources       5.12.0                   pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
interrogate               1.5.0                    pypi_0    pypi
isort                     4.3.21                   pypi_0    pypi
json-tricks               3.16.1                   pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
libffi                    3.4.2                h8ffe710_5    conda-forge
libsqlite                 3.41.2               hcfcfb64_1    conda-forge
libzlib                   1.2.13               hcfcfb64_4    conda-forge
mako                      1.2.4                    pypi_0    pypi
markdown                  3.4.3                    pypi_0    pypi
markdown-it-py            2.2.0                    pypi_0    pypi
markupsafe                2.1.2                    pypi_0    pypi
matplotlib                3.7.1                    pypi_0    pypi
mccabe                    0.7.0                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mmcv                      2.0.0                    pypi_0    pypi
mmdeploy                  1.0.0                    pypi_0    pypi
mmdeploy-runtime          1.0.0                    pypi_0    pypi
mmdeploy-runtime-gpu      1.0.0                    pypi_0    pypi
mmdet                     3.0.0                     dev_0    <develop>
mmengine                  0.7.3                    pypi_0    pypi
mmpose                    1.0.0                     dev_0    <develop>
model-index               0.1.11                   pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
multiprocess              0.70.14                  pypi_0    pypi
munkres                   1.1.4                    pypi_0    pypi
numpy                     1.24.3                   pypi_0    pypi
onnx                      1.14.0                   pypi_0    pypi
onnxruntime               1.8.1                    pypi_0    pypi
onnxruntime-gpu           1.14.1                   pypi_0    pypi
opencv-python             4.7.0.72                 pypi_0    pypi
openmim                   0.3.7                    pypi_0    pypi
openssl                   3.1.0                hcfcfb64_3    conda-forge
ordered-set               4.1.0                    pypi_0    pypi
packaging                 23.1                     pypi_0    pypi
pandas                    2.0.1                    pypi_0    pypi
parameterized             0.9.0                    pypi_0    pypi
pillow                    9.5.0                    pypi_0    pypi
pip                       23.1.2             pyhd8ed1ab_0    conda-forge
platformdirs              3.5.0                    pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
prettytable               3.7.0                    pypi_0    pypi
protobuf                  3.20.2                   pypi_0    pypi
py                        1.11.0                   pypi_0    pypi
pycocotools               2.0.6                    pypi_0    pypi
pycodestyle               2.10.0                   pypi_0    pypi
pycuda                    2022.2.2                 pypi_0    pypi
pyflakes                  3.0.1                    pypi_0    pypi
pygments                  2.15.1                   pypi_0    pypi
pyparsing                 3.0.9                    pypi_0    pypi
pyreadline3               3.4.1                    pypi_0    pypi
pytest                    7.3.1                    pypi_0    pypi
pytest-runner             6.0.0                    pypi_0    pypi
python                    3.8.16          h4de0772_1_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
pytools                   2022.1.14                pypi_0    pypi
pytz                      2023.3                   pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
regex                     2023.5.5                 pypi_0    pypi
requests                  2.30.0                   pypi_0    pypi
rich                      13.3.5                   pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
setuptools                67.7.2             pyhd8ed1ab_0    conda-forge
shapely                   2.0.1                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
sympy                     1.11.1                   pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
tensorrt                  8.2.3.0                  pypi_0    pypi
termcolor                 2.3.0                    pypi_0    pypi
terminaltables            3.1.10                   pypi_0    pypi
tk                        8.6.12               h8ffe710_0    conda-forge
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
torch                     1.12.1+cu113             pypi_0    pypi
torchaudio                0.12.1+cu113             pypi_0    pypi
torchvision               0.13.1+cu113             pypi_0    pypi
typing-extensions         4.5.0                    pypi_0    pypi
tzdata                    2023.3                   pypi_0    pypi
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
urllib3                   2.0.2                    pypi_0    pypi
vc                        14.3                hb25d44b_16    conda-forge
vc14_runtime              14.34.31931         h5081d32_16    conda-forge
vs2015_runtime            14.34.31931         hed1258a_16    conda-forge
wcwidth                   0.2.6                    pypi_0    pypi
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
xdoctest                  1.1.1                    pypi_0    pypi
xtcocotools               1.13                     pypi_0    pypi
xz                        5.4.2                h8cc25b3_0
yapf                      0.33.0                   pypi_0    pypi
zipp                      3.15.0                   pypi_0    pypi

Error traceback

**Error1**
"loading mmdeploy_trt_net.dll ...
loading mmdeploy_ort_net.dll ...
[2023-05-10 17:07:44.296] [mmdeploy] [info] [model.cpp:35] [DirectoryModel] Load model: "rtmpose-ort/rtmpose-m/"
[2023-05-10 17:07:44.297] [mmdeploy] [info] [model.cpp:35] [DirectoryModel] Load model: "rtmpose-ort/rtmdet-nano/"
2023-05-10 16:07:44.3143844 [E:onnxruntime:, provider_bridge_ort.cc:901 onnxruntime::ProviderSharedLibrary::Ensure] Failed to load library, error code: 126
[2023-05-10 17:07:44.314] [mmdeploy] [error] [ort_net.cpp:205] unhandled exception when creating ORTNet: OrtSessionOptionsAppendExecutionProvider_Cuda: Failed to load shared library
[2023-05-10 17:07:44.315] [mmdeploy] [error] [net_module.cpp:54] Failed to create Net backend: onnxruntime, config: {
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "prep_output"
  ],
  "input_map": {
    "img": "input"
  },
  "is_batched": true,
  "module": "Net",
  "name": "rtmdet",
  "output": [
    "infer_output"
  ],
  "output_map": {},
  "type": "Task"
}
[2023-05-10 17:07:44.315] [mmdeploy] [error] [task.cpp:99] error parsing config: {
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "prep_output"
  ],
  "input_map": {
    "img": "input"
  },
  "is_batched": true,
  "module": "Net",
  "name": "rtmdet",
  "output": [
    "infer_output"
  ],
  "output_map": {},
  "type": "Task"
}
[2023-05-10 17:07:44.315] [mmdeploy] [info] [inference.cpp:54] ["img"] <- ["data"]
[2023-05-10 17:07:44.315] [mmdeploy] [info] [inference.cpp:65] ["post_output"] -> ["dets"]
2023-05-10 16:07:44.3620202 [E:onnxruntime:, provider_bridge_ort.cc:901 onnxruntime::ProviderSharedLibrary::Ensure] Failed to load library, error code: 126
[2023-05-10 17:07:44.366] [mmdeploy] [error] [ort_net.cpp:205] unhandled exception when creating ORTNet: OrtSessionOptionsAppendExecutionProvider_Cuda: Failed to load shared library
[2023-05-10 17:07:44.366] [mmdeploy] [error] [net_module.cpp:54] Failed to create Net backend: onnxruntime, config: {
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "prep_output"
  ],
  "input_map": {
    "img": "input"
  },
  "is_batched": false,
  "module": "Net",
  "name": "topdownposeestimator",
  "output": [
    "infer_output"
  ],
  "output_map": {},
  "type": "Task"
}
[2023-05-10 17:07:44.366] [mmdeploy] [error] [task.cpp:99] error parsing config: {
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "prep_output"
  ],
  "input_map": {
    "img": "input"
  },
  "is_batched": false,
  "module": "Net",
  "name": "topdownposeestimator",
  "output": [
    "infer_output"
  ],
  "output_map": {},
  "type": "Task"
}
[2023-05-10 17:07:44.366] [mmdeploy] [info] [inference.cpp:54] ["img"] <- ["rois"]
[2023-05-10 17:07:44.366] [mmdeploy] [info] [inference.cpp:65] ["post_output"] -> ["keypoints"]"
**End Error1**
irexyc commented 1 year ago

For windows, if you wan't to inference by onnxruntiem-gpu, you have to copy the onnxruntime library to the mmdeploy-runtime installation directory.

You can download onnxruntime library there: https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-win-gpu-x64-1.8.1.zip

You can find mmdeploy-runtime installation directory by:

python -c 'import mmdeploy; print(mmdeploy)'
CerPhd commented 1 year ago

Thank you !!