vxgu86 commented 9 months ago

Checklist

[X] I have searched related issues but cannot get the expected help.
[X] 2. I have read the FAQ documentation but cannot get the expected help.
[X] 3. The bug has not been fixed in the latest version.

Describe the bug

想请问下 jetson上安装mmdeploy的意义是什么，是可以在jetson上转换模型吗？在jetson上安装mmdeploy后运行需要mmdet，mmdet需要mmengine，但安装只能到0.4.0，源码编译又有MMLogger的错误。

Reproduction

就是 mmdeploy.py的指令

Environment

按jetson上部署的指令安装的
MMLogger提示错误

python ./tools/deploy.py     configs/mmdet/detection/detection_tensorrt_dynamic-64x64-608x608.py     /home/user/mmdeploy_ws/deploypth/configs/rtmdet/rtmdet_l_8xb32-300e_coco.py     /home/user/mmdeploy_ws/deploypth/epoch_300.pth     /home/user/mmdeploy_ws/deploypth/mil_sea_renchuan_kr_TV_014952_20230403131233319_visible.JPG     --work-dir ../deploypth     --device cuda:0     --show
Traceback (most recent call last):
  File "./tools/deploy.py", line 335, in <module>
    main()
  File "./tools/deploy.py", line 108, in main
    logger.setLevel(log_level)
  File "/home/user/mmdeploy_ws/mmengine/mmengine/logging/logger.py", line 323, in setLevel
    logger._cache.clear()
AttributeError: 'MMLogger' object has no attribute '_cache'

Error traceback

No response

RunningLeon commented 9 months ago

hi, pls check if this PR is helpful to you: https://github.com/open-mmlab/mmdeploy/pull/2587

yinfan98 commented 9 months ago

Hi, The reason you need to build this package is that you can utilize it for model transformation as well as inference using its API.

yinfan98 commented 9 months ago

You mentioned

mmdet requires mmengine, but installs only up to 0.4.0

But I follow the guide and it install the mmengine 0.8.0. I wonder if you compiled the mmdeploy converter? If I remember correctly it installs mmengine at follow step.

Install Model Converter


# build TensorRT custom operators
mkdir -p build && cd build
cmake .. -DMMDEPLOY_TARGET_BACKENDS="trt"
make -j$(nproc) && make install

# install model converter
cd ${MMDEPLOY_DIR}
pip install -v -e .
# "-v" means verbose, or more output
# "-e" means installing a project in editable mode,
# thus any local modifications made to the code will take effect without re-installation.

ShihaoHan19980609 commented 9 months ago

我把这行代码注销掉了，不影响正常运行

613B commented 9 months ago

Hi.

If you don't need inference result for test image, it is easily to convert model on your training machine, get the onnx model, and deploy TensorRT model from onnx model on your jetson by trtexec. This way does not require mmdeploy to be installed on Jetson.

Thus, the steps are below.

On your host traning machine (that mmdeploy installed).

python ./tools/deploy.py     configs/mmdet/detection/detection_tensorrt_dynamic-64x64-608x608.py     /home/user/mmdeploy_ws/deploypth/configs/rtmdet/rtmdet_l_8xb32-300e_coco.py     /home/user/mmdeploy_ws/deploypth/epoch_300.pth     /home/user/mmdeploy_ws/deploypth/mil_sea_renchuan_kr_TV_014952_20230403131233319_visible.JPG     --work-dir ../deploypth     --device cuda:0     --show

and, you'll get ${work-dir}/end2end.onnx. (I recommend using tools/torch2onnx.py instead of tools/deploy.py to avoid the process deploying ${work-dir}/end2end.engine)

Next, place work-dir/end2end.onnx to your jetson.

On your jetson

trtexec --onnx=end2end.onnx --saveEngine=end2end.engine

Additional options may be needed for dynamic shapes, quantization and so on.

This way will be much easier. But I'm not sure if it fits your use case.

想请问下 jetson上安装mmdeploy的意义是什么

In my opinion, it helps us to avoid complex configuration about calibration when int8 quantization is needed. Also, it's nice that we can get images infer on jetson immediately.

I hope this information may help you. And I would appreciate it if you could point out any mistakes.

github-actions[bot] commented 9 months ago

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.

github-actions[bot] commented 8 months ago

This issue is closed because it has been stale for 5 days. Please open a new issue if you have similar issues or you have any new updates now.

vxgu86 commented 8 months ago

Hi.

If you don't need inference result for test image, it is easily to convert model on your training machine, get the onnx model, and deploy TensorRT model from onnx model on your jetson by trtexec. This way does not require mmdeploy to be installed on Jetson.

Thus, the steps are below.

On your host traning machine (that mmdeploy installed).
python ./tools/deploy.py     configs/mmdet/detection/detection_tensorrt_dynamic-64x64-608x608.py     /home/user/mmdeploy_ws/deploypth/configs/rtmdet/rtmdet_l_8xb32-300e_coco.py     /home/user/mmdeploy_ws/deploypth/epoch_300.pth     /home/user/mmdeploy_ws/deploypth/mil_sea_renchuan_kr_TV_014952_20230403131233319_visible.JPG     --work-dir ../deploypth     --device cuda:0     --show
and, you'll get ${work-dir}/end2end.onnx. (I recommend using tools/torch2onnx.py instead of tools/deploy.py to avoid the process deploying ${work-dir}/end2end.engine)

Next, place work-dir/end2end.onnx to your jetson.

On your jetson
trtexec --onnx=end2end.onnx --saveEngine=end2end.engine
Additional options may be needed for dynamic shapes, quantization and so on.

This way will be much easier. But I'm not sure if it fits your use case.

想请问下 jetson上安装mmdeploy的意义是什么

In my opinion, it helps us to avoid complex configuration about calibration when int8 quantization is needed. Also, it's nice that we can get images infer on jetson immediately.

I hope this information may help you. And I would appreciate it if you could point out any mistakes.

@613B Hi, thanks for your reply and sorry for my late I tried your method with /usr/src/tensorrt/bin/trtexec --onnx=/home/end2end.onnx --saveEngine=/home/work_dir/end.engine but get error as follows

`[01/04/2024-01:13:15] [W] [TRT] onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [01/04/2024-01:13:17] [I] [TRT] No importer registered for op: TRTBatchedNMS. Attempting to import as plugin. [01/04/2024-01:13:17] [I] [TRT] Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace: [01/04/2024-01:13:17] [E] [TRT] ModelImporter.cpp:773: While parsing node number 626 [TRTBatchedNMS -> "/TRTBatchedNMS_output_0"]: [01/04/2024-01:13:17] [E] [TRT] ModelImporter.cpp:774: --- Begin node --- [01/04/2024-01:13:17] [E] [TRT] ModelImporter.cpp:775: input: "/Unsqueeze_11_output_0" input: "/Sigmoid_output_0" output: "/TRTBatchedNMS_output_0" output: "/TRTBatchedNMS_output_1" name: "/TRTBatchedNMS" op_type: "TRTBatchedNMS" attribute { name: "background_label_id" i: -1 type: INT } attribute { name: "clip_boxes" i: 0 type: INT } attribute { name: "iou_threshold" f: 0.65 type: FLOAT } attribute { name: "is_normalized" i: 0 type: INT } attribute { name: "keep_topk" i: 200 type: INT } attribute { name: "num_classes" i: 24 type: INT } attribute { name: "return_index" i: 0 type: INT } attribute { name: "score_threshold" f: 0.001 type: FLOAT } attribute { name: "topk" i: 5000 type: INT } domain: "mmdeploy"

[01/04/2024-01:13:17] [E] [TRT] ModelImporter.cpp:776: --- End node --- [01/04/2024-01:13:17] [E] [TRT] ModelImporter.cpp:779: ERROR: builtin_op_importers.cpp:4870 In function importFallbackPluginImporter: [8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?" [01/04/2024-01:13:17] [E] Failed to parse onnx file [01/04/2024-01:13:17] [I] Finish parsing network model [01/04/2024-01:13:17] [E] Parsing model failed [01/04/2024-01:13:17] [E] Failed to create engine from model. [01/04/2024-01:13:17] [E] Engine set up failed ` I rechecked the operations for TensorRT custom operators as follows:

build TensorRT custom operators

mkdir -p build && cd build cmake .. -DMMDEPLOY_TARGET_BACKENDS="trt" make -j$(nproc) && make install

what is the problem?

HEIseYOUmolc commented 8 months ago

bro ，do you have more efficient ways in jetson nano？I have try the onnxruntime and tensorrt to deploy my model ，but always have problem………

onnxruntime

[2024-01-06 19:15:19.539] [mmdeploy] [info] [model.cpp:35] [DirectoryModel] Load model: "/home/nvidia/文档/mmdeploy_models/rtdetr1" [2024-01-06 19:15:19.804] [mmdeploy] [error] [resize.cpp:84] unsupported interpolation method: bicubic [2024-01-06 19:15:19.804] [mmdeploy] [error] [task.cpp:99] error parsing config: { "context": { "device": "", "model": "", "stream": "" }, "input": [ "img" ], "module": "Transform", "name": "Preprocess", "output": [ "prep_output" ], "transforms": [ { "backend_args": null, "type": "LoadImageFromFile" }, { "interpolation": "bicubic", "keep_ratio": false, "size": [ 640, 640 ], "type": "Resize" }, { "mean": [ 0, 0, 0 ], "std": [ 255, 255, 255 ], "to_rgb": true, "type": "Normalize" }, { "size_divisor": 32, "type": "Pad" }, { "type": "DefaultFormatBundle" }, { "keys": [ "img" ], "meta_keys": [ "flip", "pad_shape", "pad_param", "ori_filename", "scale_factor", "img_id", "img_path", "img_norm_cfg", "img_shape", "valid_ratio", "ori_shape", "filename", "flip_direction" ], "type": "Collect" } ], "type": "Task" } Segmentation fault (core dumped)

TensorRT

mmdeploy) nvidia@nvidia-desktop:~/文档/mmdeploy/build/bin$ ./object_detection cuda /home/nvidia/文档/mmdeploy_models/rt-detr /home/nvidia/图片/resources/test.jpg [2024-01-06 19:13:35.692] [mmdeploy] [info] [model.cpp:35] [DirectoryModel] Load model: "/home/nvidia/文档/mmdeploy_models/rt-detr" [2024-01-06 19:13:36.402] [mmdeploy] [error] [resize.cpp:84] unsupported interpolation method: bicubic [2024-01-06 19:13:36.402] [mmdeploy] [error] [task.cpp:99] error parsing config: { "context": { "device": "", "model": "", "stream": "" }, "input": [ "img" ], "module": "Transform", "name": "Preprocess", "output": [ "prep_output" ], "transforms": [ { "backend_args": null, "type": "LoadImageFromFile" }, { "interpolation": "bicubic", "keep_ratio": false, "size": [ 640, 640 ], "type": "Resize" }, { "mean": [ 0, 0, 0 ], "std": [ 255, 255, 255 ], "to_rgb": true, "type": "Normalize" }, { "size_divisor": 32, "type": "Pad" }, { "type": "DefaultFormatBundle" }, { "keys": [ "img" ], "meta_keys": [ "ori_shape", "valid_ratio", "img_norm_cfg", "pad_param", "img_path", "img_id", "img_shape", "filename", "scale_factor", "flip_direction", "ori_filename", "flip", "pad_shape" ], "type": "Collect" } ], "type": "Task" } [2024-01-06 19:13:41.757] [mmdeploy] [error] [trt_net.cpp:28] TRTNet: 6: The engine plan file is not compatible with this version of TensorRT, expecting library version 8.2.1 got 8.2.3, please rebuild. [2024-01-06 19:13:41.804] [mmdeploy] [error] [trt_net.cpp:28] TRTNet: 4: [runtime.cpp::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.) [2024-01-06 19:13:41.804] [mmdeploy] [error] [trt_net.cpp:75] failed to deserialize TRT CUDA engine [2024-01-06 19:13:41.829] [mmdeploy] [error] [net_module.cpp:54] Failed to create Net backend: tensorrt, config: { "context": { "device": "", "model": "", "stream": "" }, "input": [ "prep_output" ], "input_map": { "img": "input" }, "is_batched": false, "module": "Net", "name": "rtdetr", "output": [ "infer_output" ], "output_map": {}, "type": "Task" } [2024-01-06 19:13:41.829] [mmdeploy] [error] [task.cpp:99] error parsing config: { "context": { "device": "", "model": "", "stream": "" }, "input": [ "prep_output" ], "input_map": { "img": "input" }, "is_batched": false, "module": "Net", "name": "rtdetr", "output": [ "infer_output" ], "output_map": {}, "type": "Task" } Segmentation fault (core dumped)

613B commented 8 months ago

@vxgu86

Hi.

Probably, the cause is plugin TRTBatchedNMS being not specified. And, I apologize for giving you incorrect information. Using tools/deploy.py will takes us resolution to dependencies automatically.

I remember this problem being solved somehow before. I will rifle through the documents about that. If possible, please indicate to me the model you would like to use and its configuration files.

613B commented 8 months ago

@HEIseYOUmolc

Hi.

Where did you exec trtexec? As you may know, trtexec should be executed on the machine where you plan to use the model. So, you have to exec trtexec on your jetson nano, and convert the onnx model already got to TensorRT engine file.

HEIseYOUmolc commented 8 months ago

@HEIseYOUmolc

Hi.

Where did you exec trtexec? As you may know, trtexec should be executed on the machine where you plan to use the model. So, you have to exec trtexec on your jetson nano, and convert the onnx model already got to TensorRT engine file. @613B Thank you advice, config file to TensorRT engine file https://github.com/open-mmlab/mmdeploy/blob/2882c64eea8640f913588f6962e66abf2e7b6c86/configs/mmdet/detection/detection_tensorrt_static-640x640.py I mmdeploy in my windows computer ，and then copy file to jetson nano C++ example https://github.com/open-mmlab/mmdeploy/blob/2882c64eea8640f913588f6962e66abf2e7b6c86/demo/csrc/c/object_detection.cpp to deploy my model

File structure

--deploy.json --detail.json --end2end.engine --end2end.onnx --output_pytorch.jpg --output_tensorrt.jpg --pipeline.json

so should I use this pipeline？ copy .onnx file to jetson nano -》trtexec convert engine file -》and then how to make this engine file deploy to my object detection project If have more doc to guide ， I'll figure out what to do。

HEIseYOUmolc commented 8 months ago

@HEIseYOUmolc

Hi.

Where did you exec trtexec? As you may know, trtexec should be executed on the machine where you plan to use the model. So, you have to exec trtexec on your jetson nano, and convert the onnx model already got to TensorRT engine file. @613B

follow you advice I use the trtexec to convert my onnx file ，and get bug message：

how to fix this，thanks you help again。

(mmdeploy) nvidia@nvidia-desktop:~/文档/mmdeploy_models/rt-detr 2$ trtexec --onnx=end2end.onx --saveEngine=end2end.engine --plugins=/home/nvidia/文档/mmdeploy/mmdeploy/lib/libmmdeploy_tensorrt_ops.so --optShapes=input:1x3x640x640 &&&& RUNNING TensorRT.trtexec [TensorRT v8201] # trtexec --onnx=end2end.onx --saveEngine=end2end.engine --plugins=/home/nvidia/文档/mmdeploy/mmdeploy/lib/libmmdeploy_tensorrt_ops.so --optShapes=input:1x3x640x640 [01/07/2024-16:05:44] [I] === Model Options === [01/07/2024-16:05:44] [I] Format: ONNX [01/07/2024-16:05:44] [I] Model: end2end.onx [01/07/2024-16:05:44] [I] Output: [01/07/2024-16:05:44] [I] === Build Options === [01/07/2024-16:05:44] [I] Max batch: explicit batch [01/07/2024-16:05:44] [I] Workspace: 16 MiB [01/07/2024-16:05:44] [I] minTiming: 1 [01/07/2024-16:05:44] [I] avgTiming: 8 [01/07/2024-16:05:44] [I] Precision: FP32 [01/07/2024-16:05:44] [I] Calibration: [01/07/2024-16:05:44] [I] Refit: Disabled [01/07/2024-16:05:44] [I] Sparsity: Disabled [01/07/2024-16:05:44] [I] Safe mode: Disabled [01/07/2024-16:05:44] [I] DirectIO mode: Disabled [01/07/2024-16:05:44] [I] Restricted mode: Disabled [01/07/2024-16:05:44] [I] Save engine: end2end.engine [01/07/2024-16:05:44] [I] Load engine: [01/07/2024-16:05:44] [I] Profiling verbosity: 0 [01/07/2024-16:05:44] [I] Tactic sources: Using default tactic sources [01/07/2024-16:05:44] [I] timingCacheMode: local [01/07/2024-16:05:44] [I] timingCacheFile: [01/07/2024-16:05:44] [I] Input(s)s format: fp32:CHW [01/07/2024-16:05:44] [I] Output(s)s format: fp32:CHW [01/07/2024-16:05:44] [I] Input build shape: input=1x3x640x640+1x3x640x640+1x3x640x640 [01/07/2024-16:05:44] [I] Input calibration shapes: model [01/07/2024-16:05:44] [I] === System Options === [01/07/2024-16:05:44] [I] Device: 0 [01/07/2024-16:05:44] [I] DLACore: [01/07/2024-16:05:44] [I] Plugins: /home/nvidia/文档/mmdeploy/mmdeploy/lib/libmmdeploy_tensorrt_ops.so [01/07/2024-16:05:44] [I] === Inference Options === [01/07/2024-16:05:44] [I] Batch: Explicit [01/07/2024-16:05:44] [I] Input inference shape: input=1x3x640x640 [01/07/2024-16:05:44] [I] Iterations: 10 [01/07/2024-16:05:44] [I] Duration: 3s (+ 200ms warm up) [01/07/2024-16:05:44] [I] Sleep time: 0ms [01/07/2024-16:05:44] [I] Idle time: 0ms [01/07/2024-16:05:44] [I] Streams: 1 [01/07/2024-16:05:44] [I] ExposeDMA: Disabled [01/07/2024-16:05:44] [I] Data transfers: Enabled [01/07/2024-16:05:44] [I] Spin-wait: Disabled [01/07/2024-16:05:44] [I] Multithreading: Disabled [01/07/2024-16:05:44] [I] CUDA Graph: Disabled [01/07/2024-16:05:44] [I] Separate profiling: Disabled [01/07/2024-16:05:44] [I] Time Deserialize: Disabled [01/07/2024-16:05:44] [I] Time Refit: Disabled [01/07/2024-16:05:44] [I] Skip inference: Disabled [01/07/2024-16:05:44] [I] Inputs: [01/07/2024-16:05:44] [I] === Reporting Options === [01/07/2024-16:05:44] [I] Verbose: Disabled [01/07/2024-16:05:44] [I] Averages: 10 inferences [01/07/2024-16:05:44] [I] Percentile: 99 [01/07/2024-16:05:44] [I] Dump refittable layers:Disabled [01/07/2024-16:05:44] [I] Dump output: Disabled [01/07/2024-16:05:44] [I] Profile: Disabled [01/07/2024-16:05:44] [I] Export timing to JSON file: [01/07/2024-16:05:44] [I] Export output to JSON file: [01/07/2024-16:05:44] [I] Export profile to JSON file: [01/07/2024-16:05:44] [I] [01/07/2024-16:05:44] [I] === Device Information === [01/07/2024-16:05:44] [I] Selected Device: NVIDIA Tegra X1 [01/07/2024-16:05:44] [I] Compute Capability: 5.3 [01/07/2024-16:05:44] [I] SMs: 1 [01/07/2024-16:05:44] [I] Compute Clock Rate: 0.9216 GHz [01/07/2024-16:05:44] [I] Device Global Memory: 3962 MiB [01/07/2024-16:05:44] [I] Shared Memory per SM: 64 KiB [01/07/2024-16:05:44] [I] Memory Bus Width: 64 bits (ECC disabled) [01/07/2024-16:05:44] [I] Memory Clock Rate: 0.01275 GHz [01/07/2024-16:05:44] [I] [01/07/2024-16:05:44] [I] TensorRT version: 8.2.1 [01/07/2024-16:05:44] [I] Loading supplied plugin library: /home/nvidia/文档/mmdeploy/mmdeploy/lib/libmmdeploy_tensorrt_ops.so [01/07/2024-16:05:46] [I] [TRT] [MemUsageChange] Init CUDA: CPU +229, GPU +0, now: CPU 248, GPU 3363 (MiB) [01/07/2024-16:05:46] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 248 MiB, GPU 3363 MiB [01/07/2024-16:05:46] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 278 MiB, GPU 3393 MiB [01/07/2024-16:05:46] [I] Start parsing network model Could not open file end2end.onx Could not open file end2end.onx [01/07/2024-16:05:46] [E] [TRT] ModelImporter.cpp:735: Failed to parse ONNX model from file: end2end.onx [01/07/2024-16:05:46] [E] Failed to parse onnx file [01/07/2024-16:05:46] [I] Finish parsing network model [01/07/2024-16:05:46] [E] Parsing model failed [01/07/2024-16:05:46] [E] Failed to create engine from model. [01/07/2024-16:05:46] [E] Engine set up failed &&&& FAILED TensorRT.trtexec [TensorRT v8201] # trtexec --onnx=end2end.onx --saveEngine=end2end.engine --plugins=/home/nvidia/文档/mmdeploy/mmdeploy/lib/libmmdeploy_tensorrt_ops.so --optShapes=input:1x3x640x640

613B commented 8 months ago

@HEIseYOUmolc

Correct typo like [end2end.onx -> end2end.onnx] and try it again. Thank you.

HEIseYOUmolc commented 8 months ago

@HEIseYOUmolc

Correct typo like [end2end.onx -> end2end.onnx] and try it again. Thank you.

Thanks, very effective method, thanks for the reply

vxgu86 commented 8 months ago

hi, pls check if this PR is helpful to you: #2587

@RunningLeon Hi, my problem is concerned with mmdetection, but I saw that you do not install mmdet in the docker?

open-mmlab / mmdeploy

[Bug] jetson上安装mmdeploy的意义是？运行需要mmdet，mmdet需要mmengine，但安装只能到0.4.0，源码编译又有MMLogger的问题 #2593

Checklist

Describe the bug

Reproduction

Environment

Error traceback

Install Model Converter

On your host traning machine (that mmdeploy installed).

On your jetson

On your host traning machine (that mmdeploy installed).

On your jetson

build TensorRT custom operators

onnxruntime

TensorRT

File structure

follow you advice I use the trtexec to convert my onnx file ，and get bug message：

how to fix this，thanks you help again。