Connor923 commented 8 months ago

Checklist

[X] I have searched related issues but cannot get the expected help.
[X] 2. I have read the FAQ documentation but cannot get the expected help.
[X] 3. The bug has not been fixed in the latest version.

Describe the bug

When I converted the model to tensorRT based on the official documentation, some errors occurred. System has prompt “Could not find module 'mmdeploy_tensorrt_ops.dll' ”.

The link to the official document is as follows: https://mmdeploy.readthedocs.io/zh-cn/latest/02-how-to-run/prebuilt_package_windows.html#tensorrt

Have you ever had similar problems before, or do you have any relevant suggestions? Please let me know, thanks a lot.

Reproduction

from mmdeploy.apis import torch2onnx from mmdeploy.apis.tensorrt import onnx2tensorrt from mmdeploy.backend.sdk.export_info import export2SDK import os

img = 'mmpretrain/demo/demo.JPEG' work_dir = 'trt/resnet' save_file = 'end2end.onnx' deploy_cfg = 'mmdeploy/configs/mmpretrain/classification_tensorrt_static-224x224.py' model_cfg = 'mmpretrain/configs/resnet/resnet18_8xb32_in1k.py' model_checkpoint = 'resnet18_8xb32_in1k_20210831-fbbb1da6.pth' device = 'cuda:0'

torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, model_checkpoint, device)

onnx_model = os.path.join(work_dir, save_file) save_file = 'end2end.engine' model_id = 0 device = 'cuda:0' onnx2tensorrt(work_dir, save_file, model_id, deploy_cfg, onnx_model, device)

export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint, device=device)

Environment

Python  3.8
CUDA  V12.0
mmdeploy  1.3.1
mmdeploy-runtime  1.3.1
TensorRT-8.6.1.6

Error traceback

PS C:\ACE_Star\code> & C:/ProgramData/anaconda3/envs/openmmlab/python.exe c:/ACE_Star/code/ToTensorRT.py
01/31 20:18:03 - mmengine - WARNING - Failed to search registry with scope "mmpretrain" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. 
This may cause unexpected failure when running the built modules. Please check whether "mmpretrain" is a correct scope, or whether the registry is initialized.
01/31 20:18:03 - mmengine - WARNING - Failed to search registry with scope "mmpretrain" in the "mmpretrain_tasks" registry tree. As a workaround, the current "mmpretrain_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmpretrain" is a correct scope, or whether the registry is initialized.
Loads checkpoint by local backend from path: resnet18_8xb32_in1k_20210831-fbbb1da6.pth
01/31 20:18:05 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 
01/31 20:18:05 - mmengine - INFO - Export PyTorch model to ONNX: trt/resnet\end2end.onnx.
01/31 20:18:05 - mmengine - WARNING - Can not find torch.nn.functional._scaled_dot_product_attention, function rewrite will not be applied
01/31 20:18:05 - mmengine - INFO - Execute onnx optimize passes.
01/31 20:18:05 - mmengine - WARNING - Can not optimize model, please build torchscipt extension.
More details: https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/experimental/onnx_optimizer.md
Traceback (most recent call last):
  File "c:/ACE_Star/code/ToTensorRT.py", line 23, in <module>
    onnx2tensorrt(work_dir, save_file, model_id, deploy_cfg, onnx_model, device)
  File "C:\ProgramData\anaconda3\envs\openmmlab\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 356, in _wrap
    return self.call_function(func_name_, *args, **kwargs)
  File "C:\ProgramData\anaconda3\envs\openmmlab\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 326, in call_function
    return self.call_function_local(func_name, *args, **kwargs)
  File "C:\ProgramData\anaconda3\envs\openmmlab\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 275, in call_function_local
    return pipe_caller(*args, **kwargs)
  File "C:\ProgramData\anaconda3\envs\openmmlab\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "C:\ProgramData\anaconda3\envs\openmmlab\lib\site-packages\mmdeploy\backend\tensorrt\onnx2tensorrt.py", line 79, in onnx2tensorrt
    from_onnx(
  File "C:\ProgramData\anaconda3\envs\openmmlab\lib\site-packages\mmdeploy\backend\tensorrt\utils.py", line 157, in from_onnx
    load_tensorrt_plugin()
  File "C:\ProgramData\anaconda3\envs\openmmlab\lib\site-packages\mmdeploy\backend\tensorrt\init_plugins.py", line 33, in load_tensorrt_plugin
    ctypes.CDLL(lib_path)
  File "C:\ProgramData\anaconda3\envs\openmmlab\lib\ctypes\__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'C:\ProgramData\anaconda3\envs\openmmlab\lib\site-packages\mmdeploy\lib\mmdeploy_tensorrt_ops.dll' (or one of its dependencies). Try using the full path with constructor syntax.

xzh929 commented 8 months ago

you can run tools/check_env.py,if tensorrt custom ops is not Available,you need to build it.See how to build[docs/en/01-how-to-build/windows.md](https://github.com/open-mmlab/mmdeploy/blob/main/docs/en/01-how-to-build/windows.md)

Ultraopxt commented 4 months ago

you can run tools/check_env.py,if tensorrt custom ops is not Available,you need to build it.See how to build[docs/en/01-how-to-build/windows.md](https://github.com/open-mmlab/mmdeploy/blob/main/docs/en/01-how-to-build/windows.md)

I meet the same error, and i run python tools/check_env.py ,got the following:

05/24 15:07:53 - mmengine - INFO -

05/24 15:07:53 - mmengine - INFO - **Environmental information** 05/24 15:07:57 - mmengine - INFO - sys.platform: win32 05/24 15:07:57 - mmengine - INFO - Python: 3.10.14 | packaged by Anaconda, Inc. | (main, May 6 2024, 19:44:50) [MSC v.1916 64 bit (AMD64)] 05/24 15:07:57 - mmengine - INFO - CUDA available: True 05/24 15:07:57 - mmengine - INFO - MUSA available: False 05/24 15:07:57 - mmengine - INFO - numpy_random_seed: 2147483648 05/24 15:07:57 - mmengine - INFO - GPU 0: NVIDIA GeForce RTX 4090 05/24 15:07:57 - mmengine - INFO - CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8 05/24 15:07:57 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.8, V11.8.89 05/24 15:07:57 - mmengine - INFO - MSVC: 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.37.32824 版 05/24 15:07:57 - mmengine - INFO - GCC: n/a 05/24 15:07:57 - mmengine - INFO - PyTorch: 2.3.0+cu118 05/24 15:07:57 - mmengine - INFO - PyTorch compiling details: PyTorch built with:

C++ Version: 201703
MSVC 192930151
Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361)
OpenMP 2019
LAPACK is enabled (usually provided by MKL)
CPU capability usage: AVX2
CUDA Runtime 11.8
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37
CuDNN 8.7
Magma 2.5.4
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=C:/actions-runner/_work/pytorch/pytorch/builder/windows/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /Zc:__cplusplus /bigobj /FS /utf-8 -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 /wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 /wd4804 /wd4273, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.3.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

05/24 15:07:57 - mmengine - INFO - TorchVision: 0.18.0+cu118 05/24 15:07:57 - mmengine - INFO - OpenCV: 4.9.0 05/24 15:07:57 - mmengine - INFO - MMEngine: 0.10.4 05/24 15:07:57 - mmengine - INFO - MMCV: 2.1.0 05/24 15:07:57 - mmengine - INFO - MMCV Compiler: MSVC 193732824 05/24 15:07:57 - mmengine - INFO - MMCV CUDA Compiler: 11.8 05/24 15:07:57 - mmengine - INFO - MMDeploy: 1.3.1+ec4abe5 05/24 15:07:57 - mmengine - INFO -

05/24 15:07:57 - mmengine - INFO - **Backend information** 05/24 15:07:57 - mmengine - INFO - tensorrt: 10.0.1 05/24 15:07:57 - mmengine - INFO - tensorrt custom ops: Available 05/24 15:07:57 - mmengine - INFO - ONNXRuntime: 1.12.0 05/24 15:07:57 - mmengine - INFO - ONNXRuntime-gpu: None 05/24 15:07:57 - mmengine - INFO - ONNXRuntime custom ops: Available 05/24 15:07:57 - mmengine - INFO - pplnn: None 05/24 15:07:57 - mmengine - INFO - ncnn: None 05/24 15:07:57 - mmengine - INFO - snpe: None 05/24 15:07:57 - mmengine - INFO - openvino: None 05/24 15:07:57 - mmengine - INFO - torchscript: 2.3.0+cu118 05/24 15:07:57 - mmengine - INFO - torchscript custom ops: NotAvailable 05/24 15:07:57 - mmengine - INFO - rknn-toolkit: None 05/24 15:07:57 - mmengine - INFO - rknn-toolkit2: None 05/24 15:07:57 - mmengine - INFO - ascend: None 05/24 15:07:57 - mmengine - INFO - coreml: None 05/24 15:07:57 - mmengine - INFO - tvm: None 05/24 15:07:57 - mmengine - INFO - vacc: None 05/24 15:07:57 - mmengine - INFO -

05/24 15:07:57 - mmengine - INFO - **Codebase information** 05/24 15:07:57 - mmengine - INFO - mmdet: 3.3.0 05/24 15:07:57 - mmengine - INFO - mmseg: None 05/24 15:07:57 - mmengine - INFO - mmpretrain: 1.2.0 05/24 15:07:57 - mmengine - INFO - mmocr: None 05/24 15:07:57 - mmengine - INFO - mmagic: None 05/24 15:07:57 - mmengine - INFO - mmdet3d: None 05/24 15:07:57 - mmengine - INFO - mmpose: None 05/24 15:07:57 - mmengine - INFO - mmrotate: None 05/24 15:07:57 - mmengine - INFO - mmaction: None 05/24 15:07:57 - mmengine - INFO - mmrazor: None 05/24 15:07:57 - mmengine - INFO - mmyolo: None

Jellie122 commented 2 months ago

Hi, @Ultraopxt I just encountered this error too. Did u solve this problem? Hope to receive ur reply.

rlewkowicz commented 2 months ago

I think this is the tensorrt versions. Lots of things dont support 10 I think. I'm still working it out though.

rlewkowicz commented 2 months ago

Yeah, for me it was just use 8

Jellie122 commented 2 months ago

Yeah, for me it was just use 8

Thanks for ur reply! I previouslty using tensorrt 8 instead of 10, but i got another error: TypeError: pybind11::init(): factory function returned nullptr

when i runned the following code:

def build_engine(model_file):

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network,trt.OnnxParser(network, TRT_LOGGER) as parser:
    builder.max_workspace_size = MAX_WORKSPACE_SIZE
    builder.max_batch_size = MAX_BATCH_SIZE
    with open(model_file, 'rb') as model:
        parser.parse(model.read())
        engine = builder.build_cuda_engine(network)
return engine

Have u run into this error?

Jellie122 commented 2 months ago

@rlewkowicz can u share ur packages and version of ur virtual env? Thanks!

rlewkowicz commented 2 months ago

Do note, that I think like ultralytics and some others use paths under the hood. So like I have the cuda toolkit, and I dump cudnn, tensort all into that folder, and then just for the dlls I copy lib to bin. They just load system dlls and don't use any python specifics at all. So cudatoolkit, dump all other nvidia assets in there, add it to path, success.

Package                        Version
------------------------------ --------------------
absl-py                        1.3.0
addict                         2.4.0
aiofiles                       23.2.1
albumentations                 1.4.2
altair                         5.3.0
annotated-types                0.7.0
anyio                          4.4.0
appdirs                        1.4.4
asgiref                        3.8.1
asttokens                      2.2.1
attr                           0.3.1
attrs                          23.2.0
azure-core                     1.30.2
azure-storage-blob             12.20.0
backcall                       0.2.0
beautifulsoup4                 4.12.3
bleach                         5.0.1
boto                           2.49.0
boto3                          1.34.142
botocore                       1.34.142
boxing                         0.1.4
cachetools                     5.2.0
certifi                        2024.2.2
cffi                           1.16.0
charset-normalizer             3.3.2
click                          8.1.7
colorama                       0.4.6
coloredlogs                    15.0.1
contourpy                      1.2.0
cryptography                   42.0.8
cycler                         0.12.1
decorator                      5.1.1
defusedxml                     0.7.1
distro                         1.9.0
Django                         3.2.25
django-annoying                0.10.6
django-cors-headers            3.6.0
django-csp                     3.7
django-debug-toolbar           3.2.1
django-environ                 0.10.0
django-extensions              3.1.0
django-filter                  2.4.0
django-model-utils             4.1.1
django-ranged-fileresponse     0.1.2
django-rq                      2.5.1
django-storages                1.12.3
django-user-agents             0.4.0
djangorestframework            3.13.1
drf-dynamic-fields             0.3.0
drf-flex-fields                0.9.5
drf-generators                 0.3.0
ecal5                          5.11.1
exceptiongroup                 1.2.1
executing                      1.2.0
expiringdict                   1.2.2
fastapi                        0.112.0
ffmpy                          0.4.0
filelock                       3.13.1
flatbuffers                    24.3.25
fonttools                      4.49.0
fsspec                         2024.2.0
gdown                          5.2.0
gitdb                          4.0.10
GitPython                      3.1.30
google-api-core                2.19.1
google-auth                    2.32.0
google-auth-oauthlib           0.4.6
google-cloud-appengine-logging 1.4.4
google-cloud-audit-log         0.2.5
google-cloud-core              2.4.1
google-cloud-logging           3.10.0
google-cloud-storage           2.17.0
google-crc32c                  1.5.0
google-resumable-media         2.7.1
googleapis-common-protos       1.63.2
gradio                         4.31.5
gradio_client                  0.16.4
grpc-google-iam-v1             0.13.1
grpcio                         1.64.1
grpcio-status                  1.62.2
h11                            0.14.0
htmlmin                        0.1.12
httpcore                       1.0.5
httpx                          0.27.0
huggingface-hub                0.23.2
humanfriendly                  10.0
humansignal-drf-yasg           1.21.9
idna                           3.6
ijson                          3.2.3
imageio                        2.34.0
imgviz                         1.7.5
importlib_resources            6.4.0
inflection                     0.5.1
ipython                        8.8.0
isodate                        0.6.1
jedi                           0.18.2
Jinja2                         3.1.3
jmespath                       1.0.1
joblib                         1.3.2
jsonschema                     3.2.0
kiwisolver                     1.4.5
label-studio                   1.12.1
label-studio-converter         0.0.58
label-studio-tools             0.0.4
labelme                        5.4.1
launchdarkly-server-sdk        8.2.1
lazy_loader                    0.3
lockfile                       0.12.2
lxml                           5.2.2
Markdown                       3.4.1
markdown-it-py                 3.0.0
MarkupSafe                     2.1.5
matplotlib                     3.8.3
matplotlib-inline              0.1.6
mdurl                          0.1.2
mpmath                         1.3.0
natsort                        8.4.0
networkx                       3.2.1
nltk                           3.6.7
numpy                          1.26.4
oauthlib                       3.2.2
onnx                           1.14.0
onnx-simplifier                0.4.36
onnxruntime                    1.15.1
onnxruntime-gpu                1.18.0
onnxsim                        0.4.36
onnxslim                       0.1.31
openai                         1.35.12
opencv-python                  4.9.0.80
opencv-python-headless         4.9.0.80
ordered-set                    4.0.2
orjson                         3.10.6
packaging                      23.2
pandas                         2.2.1
parso                          0.8.3
pickleshare                    0.7.5
Pillow                         9.5.0
pip                            22.3.1
prompt-toolkit                 3.0.36
proto-plus                     1.24.0
protobuf                       3.20.3
psutil                         5.9.8
psycopg2-binary                2.9.9
pure-eval                      0.2.2
py-cpuinfo                     9.0.0
pyasn1                         0.4.8
pyasn1-modules                 0.2.8
pycocotools                    2.0.7
pycparser                      2.22
pydantic                       2.8.2
pydantic_core                  2.20.1
pydub                          0.25.1
Pygments                       2.14.0
pyparsing                      3.1.2
PyQt5                          5.15.10
PyQt5-Qt5                      5.15.2
PyQt5-sip                      12.13.0
pyreadline3                    3.4.1
pyRFC3339                      1.1
pyrsistent                     0.20.0
PySocks                        1.7.1
python-dateutil                2.9.0.post0
python-json-logger             2.0.4
python-multipart               0.0.9
pytz                           2022.7.1
PyYAML                         6.0.1
QtPy                           2.4.1
rapidfuzz                      3.5.2
redis                          3.5.3
regex                          2024.5.15
requests                       2.31.0
requests-oauthlib              1.3.1
rich                           13.7.1
rq                             1.10.1
rsa                            4.9
ruff                           0.5.6
rules                          2.2
s3transfer                     0.10.2
safetensors                    0.4.3
scikit-image                   0.22.0
scikit-learn                   1.4.1.post1
scipy                          1.13.0
screen-ocr                     0.5.0
seaborn                        0.13.2
semantic-version               2.10.0
semver                         3.0.2
sentry-sdk                     2.8.0
setuptools                     65.5.0
shellingham                    1.5.4
six                            1.16.0
smmap                          5.0.0
sniffio                        1.3.1
soupsieve                      2.5
sqlparse                       0.5.0
stack-data                     0.6.2
starlette                      0.37.2
sympy                          1.12
tensorboard                    2.11.0
tensorboard-data-server        0.6.1
tensorboard-plugin-wit         1.8.1
tensorrt                       8.5.2.2
termcolor                      2.4.0
thop                           0.1.1.post2209072238
threadpoolctl                  3.4.0
tifffile                       2024.2.12
tomlkit                        0.12.0
toolz                          0.12.1
torch                          2.1.2+cu118
torchaudio                     2.1.2+cu118
torchvision                    0.16.2+cu118
tqdm                           4.66.2
traitlets                      5.8.0
typer                          0.12.3
typing_extensions              4.10.0
tzdata                         2024.1
ua-parser                      0.18.0
ujson                          5.10.0
ultralytics                    8.1.34
uritemplate                    4.1.1
urllib3                        2.2.2
user-agents                    2.2.0
uvicorn                        0.30.5
wcwidth                        0.2.5
webencodings                   0.5.1
websockets                     11.0.3
Werkzeug                       2.2.2
wheel                          0.38.4
windows-capture                1.0.51
winsdk                         1.0.0b10
xmljson                        0.2.0

rlewkowicz commented 2 months ago

@Jellie122 ^^

Jellie122 commented 2 months ago

@rlewkowicz OMG!!!! It works, thank u very much! I've been stuck on this error for a week! And i can't find the solutions from issures. (tear of joy.jpg)

rlewkowicz commented 2 months ago

@rlewkowicz OMG!!!! It works, thank u very much! I've been stuck on this error for a week! And i can't find the solutions from issures. (tear of joy.jpg)

The esoteric dependency management is the hardest part LOL. Windows is so hard to work on too. If you can I find linux easier my self for most stuff.

Jellie122 commented 2 months ago

Yeah, i think so too. Many packages only work on linux, and many bugs only show up on windows. Most of people use linux for development. There are few references.

Jellie122 commented 2 months ago

@rlewkowicz Anyway, thank u for ur help! :)

maochenyu1 commented 2 days ago

Hi,I just encountered this error too. Did u solve this problem? Hope to receive ur reply.

Jellie122 commented 1 day ago

Hi,I just encountered this error too. Did u solve this problem? Hope to receive ur reply.

Yes. You can refer to our conversation above. Make sure ur DLLs are copied into bin.

maochenyu1 commented 38 minutes ago

Thanks for ur reply! Do you mean copying the Dlls from mmdeploy/lib into the CUDA bin?I tried but it failed

open-mmlab / mmdeploy

[Bug] Could not find module 'mmdeploy_tensorrt_ops.dll' #2664

Checklist

Describe the bug

Reproduction

Environment

Error traceback

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)