search

Project-MONAI / MONAI

AI Toolkit for Healthcare Imaging
https://monai.io/
Apache License 2.0
5.91k stars 1.09k forks source link

Target GPU SM 70 is not supported by TensorRT 10.5+ #8198

Closed KumoLiu closed 1 week ago

KumoLiu commented 1 week ago 
[2024-11-11T23:29:37.758Z] ======================================================================

[2024-11-11T23:29:37.758Z] ERROR: test_trt_export_3_fp16 (tests.test_bundle_trt_export.TestTRTExport)

[2024-11-11T23:29:37.758Z] ----------------------------------------------------------------------

[2024-11-11T23:29:37.758Z] Traceback (most recent call last):

[2024-11-11T23:29:37.758Z]   File "/opt/monai/tests/utils.py", line 810, in command_line_tests

[2024-11-11T23:29:37.759Z]     normal_out = subprocess.run(cmd, env=test_env, check=True, capture_output=True)

[2024-11-11T23:29:37.759Z]   File "/usr/lib/python3.10/subprocess.py", line 526, in run

[2024-11-11T23:29:37.759Z]     raise CalledProcessError(retcode, process.args,

[2024-11-11T23:29:37.759Z] subprocess.CalledProcessError: Command '['python', '-m', 'monai.bundle', 'trt_export', 'network_def', '--filepath', '/tmp/tmpiha_eyf8/model_trt_fp16.ts', '--meta_file', '/opt/monai/tests/testing_data/metadata.json', '--config_file', "['/opt/monai/tests/testing_data/inference.json','/tmp/tmpiha_eyf8/def_args.yaml']", '--ckpt_file', '/tmp/tmpiha_eyf8/model.pt', '--args_file', '/tmp/tmpiha_eyf8/def_args.yaml', '--precision', 'fp16', '--input_shape', '[1, 1, 96, 96, 96]', '--dynamic_batch', '[1, 4, 8]']' returned non-zero exit status 1.

[2024-11-11T23:29:37.759Z] 

[2024-11-11T23:29:37.759Z] The above exception was the direct cause of the following exception:

[2024-11-11T23:29:37.759Z] 

[2024-11-11T23:29:37.759Z] Traceback (most recent call last):

[2024-11-11T23:29:37.759Z]   File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func

[2024-11-11T23:29:37.759Z]     return func(*(a + p.args), **p.kwargs, **kw)

[2024-11-11T23:29:37.759Z]   File "/opt/monai/tests/test_bundle_trt_export.py", line 88, in test_trt_export

[2024-11-11T23:29:37.759Z]     command_line_tests(cmd)

[2024-11-11T23:29:37.759Z]   File "/opt/monai/tests/utils.py", line 816, in command_line_tests

[2024-11-11T23:29:37.759Z]     raise RuntimeError(f"subprocess call error {e.returncode}: {errors}, {output}") from e

[2024-11-11T23:29:37.759Z] RuntimeError: subprocess call error 1: b'WARNING:root:Given dtype that does not have direct mapping to torch (dtype.unknown), defaulting to torch.float

[2024-11-11T23:29:37.759Z] WARNING:root:Given dtype that does not have direct mapping to torch (dtype.unknown), defaulting to torch.float

[2024-11-11T23:29:37.759Z] WARNING: [Torch-TensorRT] - Detected and removing exception in TorchScript IR for node:  = prim::If(%387) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:562:8  block0():    %388 : str = aten::format(%318, %386) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:29     = prim::RaiseException(%388, %317) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:12    -> ()  block1():    -> ()

[2024-11-11T23:29:37.759Z] WARNING: [Torch-TensorRT] - Detected and removing exception in TorchScript IR for node:  = prim::If(%401) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:562:8  block0():    %402 : str = aten::format(%318, %400) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:29     = prim::RaiseException(%402, %317) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:12    -> ()  block1():    -> ()

[2024-11-11T23:29:37.759Z] WARNING: [Torch-TensorRT] - Detected and removing exception in TorchScript IR for node:  = prim::If(%415) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:562:8  block0():    %416 : str = aten::format(%318, %414) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:29     = prim::RaiseException(%416, %317) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:12    -> ()  block1():    -> ()

[2024-11-11T23:29:37.759Z] WARNING: [Torch-TensorRT] - Detected and removing exception in TorchScript IR for node:  = prim::If(%429) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:562:8  block0():    %430 : str = aten::format(%318, %428) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:29     = prim::RaiseException(%430, %317) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:12    -> ()  block1():    -> ()

[2024-11-11T23:29:37.759Z] WARNING: [Torch-TensorRT] - Conv3d layer with kernel size = 1 configuration incurs a failure with TensorRT tactic optimizer in some cases.     Github issue: https://github.com/pytorch/TensorRT/issues/1445. Other conv variants do not have this issue.

[2024-11-11T23:29:37.759Z] ERROR: [Torch-TensorRT TorchScript Conversion Context] - IBuilder::buildSerializedNetwork: Error Code 9: API Usage Error (Target GPU SM 70 is not supported by this TensorRT release.)