Unable to use any Torch-TensorRT methods

[Arjunp24]

I'm facing this error:

AttributeError: module 'torch_tensorrt' has no attribute 'compile'

I also get this error when I try to use any other method like Input().

This is how I installed Torch-TensorRT: pip install torch-tensorrt -f github.com/NVIDIA/Torch-TensorRT/releases

Code (from official documentation):

import torch_tensorrt

model = model.eval()
compile_settings = {
    "input_shapes": [
        {
            "min": [1, 1, 16, 16],
            "opt": [1, 1, 32, 32],
            "max": [1, 1, 64, 64]
        },
    ],
    "op_precision": torch.half # Run with fp16
}
enabled_precisions = {torch.float, torch.half}

trt_ts_module = torch_tensorrt.compile(model, inputs=compile_settings, enabled_precisions=enabled_precisions) 

Stack Trace:

AttributeError                            Traceback (most recent call last)
<command-3167120371910218> in <module>
     14 enabled_precisions = {torch.float, torch.half}
     15 
---> 16 trt_ts_module = torch_tensorrt.compile(model, inputs=compile_settings, enabled_precisions=enabled_precisions)

AttributeError: module 'torch_tensorrt' has no attribute 'compile'

Please let me know how I can fix this issue.

[narendasan]

What happens if you do something like dir(torch_tensorrt)?

[Arjunp24]

Hi @narendasan , this is what I get:

['__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__']

[narendasan]

Ah I think I figured out what happened. You have a malformed url.

 pip install torch-tensorrt -f github.com/NVIDIA/Torch-TensorRT/releases
                               ^

It should be

pip3 install torch-tensorrt -f https://github.com/NVIDIA/Torch-TensorRT/releases

What is happening is on pypi we have a package which we registered but cant currently use due to ABI rules for compiled python packages. Because your url is incorrect it is fetching that placeholder package instead of the build package from our release page.

[narendasan]

Easy way to verify that it is correct is by making sure the installed version is 1.0.0 not 0.0.0

[Arjunp24]

I'm getting the same error after using the correct url. The output to dir(torch_tensorrt) also remains the same as what I got (https://github.com/NVIDIA/Torch-TensorRT/issues/733#issuecomment-974486094).

[narendasan]

I'd recommend just downloading the appropriate wheel file and installing directly. If your python version isn't between 3.6-3.9 that might cause the issue too.

[koflp]

Hi dear Team, currently, with pip3 install torch-tensorrt -f https://github.com/NVIDIA/Torch-TensorRT/releases, I still get v 0.0.0, which is just the placeholder. How can I complete the installation on WIndows?

[narendasan]

We don't have prebuilt packages for Windows. You will need to build from source. The library had unofficial support a while back but it hasn't been too well maintained. You can try and file issues where you hit problems and we can help as much as we can

[pkarimib]

@narendasan Where can I find the appropriate wheel file for Jetson Nano? Currently, my TensorRT version is shown as 0.0.0.

System information:

NVIDIA Jetson Nano (Developer Kit Version)
 L4T 32.6.1 [ JetPack 4.6 ]
   Ubuntu 18.04.6 LTS
   Kernel Version: 4.9.253-tegra
 CUDA 10.2.300
   CUDA Architecture: 5.3
 OpenCV version: 4.1.1
   OpenCV Cuda: NO
 CUDNN: 8.2.1.32
 TensorRT: 8.0.1.6
 Vision Works: 1.6.0.501
 VPI: ii libnvvpi1 1.1.15 arm64 NVIDIA Vision Programming Interface library
 Vulcan: 1.2.70

[narendasan]

For Jetson you will need to build from source for now: https://nvidia.github.io/Torch-TensorRT/tutorials/installation.html#building-natively-on-aarch64-jetson

[pkarimib]

Thank you so much for your quick response. I tried going through the instructions, but I get the following error:

Command:

bazel build //:libtorchtrt -c opt --distdir thrid_party/distdir/aarch64-linux-gnu

Error:

DEBUG: /home/nms/.cache/bazel/_bazel_nms/273a2ba260e6c622088d79767b1ce85d/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /home/nms/.cache/bazel/_bazel_nms/273a2ba260e6c622088d79767b1ce85d/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:14: rbe_ubuntu1804_java11 not using checked in configs as detect_java_home was set to True
DEBUG: /home/nms/.cache/bazel/_bazel_nms/273a2ba260e6c622088d79767b1ce85d/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /home/nms/.cache/bazel/_bazel_nms/273a2ba260e6c622088d79767b1ce85d/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:14: rbe_ubuntu1604_java8 not using checked in configs as detect_java_home was set to True
ERROR: Skipping '//:libtorch': no such target '//:libtorch': target 'libtorch' not declared in package '' defined by /home/nms/bazel/BUILD
WARNING: Target pattern parsing failed.
ERROR: no such target '//:libtorch': target 'libtorch' not declared in package '' defined by /home/nms/bazel/BUILD
INFO: Elapsed time: 0.285s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

I have added the following to the WORKSPACE:

new_local_repository(
    name = "cudnn",
    path = "/usr/",
    build_file = "@//third_party/cudnn/local:BUILD"
)

new_local_repository(
name = "tensorrt",
path = "/usr/",
build_file = "@//third_party/tensorrt/local:BUILD"
)

new_local_repository(
    name = "libtorch",
    path = "/usr/local/lib/python3.6/dist-packages/torch",
    build_file = "third_party/libtorch/BUILD"
)

new_local_repository(
    name = "libtorch_pre_cxx11_abi",
    path = "/usr/local/lib/python3.6/dist-packages/torch",
    build_file = "third_party/libtorch/BUILD"
)

Do you know what could be the issue? Also, in the instructions, it says I could run python3 setup.py bdist_wheel –use-cxx11-abi to install, but I'm not sure which setup.py file it means.

Thanks a lot!

[Hommus]

@pkarimib I'm not sure about the bazel issue, but I believe the setup.py file is at Torch-TensorRT/py/setup.py, and you might have to add the paths for cudnn and tensorrt.

[github-actions[bot]]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

[github-actions[bot]]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

[jaehyun-ko]

I think you have torch_tensort.py file in your working directory.