search

Project-MONAI / MONAI

AI Toolkit for Healthcare Imaging
https://monai.io/
Apache License 2.0
5.81k stars 1.07k forks source link

OptionalImportError in test_trt_compile #8066

Closed KumoLiu closed 1 month ago

KumoLiu commented 1 month ago 

[2024-09-04T18:48:36.293Z] ======================================================================

[2024-09-04T18:48:36.293Z] ERROR: test_handler (tests.test_trt_compile.TestTRTCompile)

[2024-09-04T18:48:36.293Z] ----------------------------------------------------------------------

[2024-09-04T18:48:36.293Z] Traceback (most recent call last):

[2024-09-04T18:48:36.293Z]   File "/tmp/tmp.6HC8gPAe2T/tests/test_trt_compile.py", line 66, in test_handler

[2024-09-04T18:48:36.293Z]     net1.forward(torch.tensor([[0.0, 1.0], [1.0, 2.0]], device="cuda"))

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 496, in trt_forward

[2024-09-04T18:48:36.293Z]     return self._trt_compiler.forward(self, argv, kwargs)

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 351, in forward

[2024-09-04T18:48:36.293Z]     raise e

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 342, in forward

[2024-09-04T18:48:36.293Z]     self._build_and_save(model, build_args)

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 446, in _build_and_save

[2024-09-04T18:48:36.293Z]     tt_inputs = [get_torch_trt_input(i.shape, self.dynamic_batchsize) for i in inputs]

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 446, in <listcomp>

[2024-09-04T18:48:36.293Z]     tt_inputs = [get_torch_trt_input(i.shape, self.dynamic_batchsize) for i in inputs]

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 442, in get_torch_trt_input

[2024-09-04T18:48:36.293Z]     return torch_tensorrt.Input(

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/monai/utils/module.py", line 444, in __getattr__

[2024-09-04T18:48:36.293Z]     raise self._exception

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/monai/utils/module.py", line 402, in optional_import

[2024-09-04T18:48:36.293Z]     pkg = __import__(module)  # top level module

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/__init__.py", line 114, in <module>

[2024-09-04T18:48:36.293Z]     _register_with_torch()

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/__init__.py", line 107, in _register_with_torch

[2024-09-04T18:48:36.293Z]     torch.ops.load_library(linked_file_full_path)

[2024-09-04T18:48:36.293Z]   File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1295, in load_library

[2024-09-04T18:48:36.293Z]     ctypes.CDLL(path)

[2024-09-04T18:48:36.293Z]   File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__

[2024-09-04T18:48:36.293Z]     self._handle = _dlopen(self._name, mode)

[2024-09-04T18:48:36.293Z] monai.utils.module.OptionalImportError: import torch_tensorrt (/usr/local/lib/python3.10/dist-packages/torch_tensorrt/lib/libtorchtrt.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE).

[2024-09-04T18:48:36.293Z] 

[2024-09-04T18:48:36.293Z] For details about installing the optional dependencies, please visit:

[2024-09-04T18:48:36.293Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
KumoLiu commented 1 month ago

Hi @borisfom and @binliunls, could you please help take a look at the issue? Thanks.

borisfom commented 1 month ago

As I noticed in Slack thread: That issue looks like the environment is corrupted (torch_tensorrt / torch versions mismatch): [2024-09-04T18:48:36.293Z] monai.utils.module.OptionalImportError: import torch_tensorrt (/usr/local/lib/python3.10/dist-packages/torch_tensorrt/lib/libtorchtrt.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE). This may happen if during setup, either torch or torch_tensorrt package is upgraded.

KumoLiu commented 1 month ago

Thanks @borisfom, fixed it in the ci env.