Can please somebody give a clear explanation of how to install torch-tensorrt on Windows?

❓ Question

Hello,

I've encountered problems installing torch-tensorrt on Windows 10

No matter how I try, how many sources I look up to, there is no clear explanation on how to do everything. The documentation is vague, and because I am used to working with python code, which does everything for you, that is pip install... python code.py, and nothing more is required, I do not have as much experience with cmake, building libraries, files, and c++, which makes it very difficult to follow along the installation process.

Now I've tried to follow along instructions from the main page

pip install torch-tensorrt doesn't work downloaded zip file of this repository; python setup.py install also doesn't work

installed bazel modified the workspace, still nothing

tried to directly import into code py/torch-tensorrt - nothing

then inside the py folder opened command prompt ant typed in:

bazel build //:libtorchtrt --compilation_mode=dbg

and received this error:

`Starting local Bazel server and connecting to it... INFO: Repository libtorch instantiated at: D:/pyth/tensorrt-main/WORKSPACE:53:13: in Repository rule http_archive defined at: C:/users/tomas/_bazel_tomas/r4zfvyvs/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in WARNING: Download from https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-latest.zip failed: class com.google.devtools.build.lib.bazel.repository.downloader.ContentLengthMismatchException Bytes read 2210658461 but wanted 2501377827 ERROR: An error occurred during the fetch of repository 'libtorch': Traceback (most recent call last): File "C:/users/tomas/_bazel_tomas/r4zfvyvs/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl download_info = ctx.download_and_extract( Error in download_and_extract: java.io.IOException: Error downloading [https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-latest.zip] to C:/users/tomas/_bazel_tomas/r4zfvyvs/external/libtorch/temp7217651597570855917/libtorch-cxx11-abi-shared-with-deps-latest.zip: Bytes read 2210658461 but wanted 2501377827 ERROR: D:/pyth/tensorrt-main/WORKSPACE:53:13: fetching http_archive rule //external:libtorch: Traceback (most recent call last): File "C:/users/tomas/_bazel_tomas/r4zfvyvs/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl download_info = ctx.download_and_extract( Error in download_and_extract: java.io.IOException: Error downloading [https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-latest.zip] to C:/users/tomas/_bazel_tomas/r4zfvyvs/external/libtorch/temp7217651597570855917/libtorch-cxx11-abi-shared-with-deps-latest.zip: Bytes read 2210658461 but wanted 2501377827 ERROR: D:/pyth/tensorrt-main/core/util/logging/BUILD:13:11: //core/util/logging:logging depends on @libtorch//:libtorch in repository @libtorch which failed to fetch. no such package '@libtorch//': java.io.IOException: Error downloading [https://download.pytorch.org/libtorch/nightly/cu121/libtorch-cxx11-abi-shared-with-deps-latest.zip] to C:/users/tomas/_bazel_tomas/r4zfvyvs/external/libtorch/temp7217651597570855917/libtorch-cxx11-abi-shared-with-deps-latest.zip: Bytes read 2210658461 but wanted 2501377827 ERROR: Analysis of target '//:libtorchtrt' failed; build aborted: INFO: Elapsed time: 458.697s INFO: 0 processes. FAILED: Build did NOT complete successfully (64 packages loaded, 413 targets configured) Fetching https://download.pytorch.org/...orch-cxx11-abi-shared-with-deps-latest.zip; 2.1 GiB (2,210,121,825B) 446s

And also tried some other things, I cannot remember, but unsuccessfully.

THANK YOU FOR YOUR HELP IN ADVANCE

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

PyTorch Version: 2.1.2+cu121
OS : Windows 10
I am running python and pytorch straight from Windows, without any environment
Python version: 3.10.13
CUDA version: 12.1 update 1
GPU models and configuration: GTX 1660 TI

Hello - we are working on a pip-installable Windows distribution for which a sample can be found on the windows_CI branch. This branch currently only supports our Dynamo paths (ir=dynamo, ir=torch_compile) and a limited TensorRT runtime (though #1943 will improve this runtime constraint). It does not require bazel to install.

In order to use it, the Torch distribution requires this patch: https://github.com/pytorch/pytorch/pull/111313, to enable Dynamo experimentally on Windows, since Torch does not yet fully support Dynamo on Windows. Then, once cloned, running python setup.py install from the root of the directory would begin the package installation. I have tested it successfully on Python 3.10.

Hello again, thank you for your reply, @gs-olive, and I apologize, for replying so late. Thank you for pointing me to the windows_CI branch, I haven't noticed it then.

Now if I use git clone https://github.com/pytorch/TensorRT.git (even though this is the same link to non windows_CI, it's on windows_CI branch) and run python setup.py install I get Could not find bazel in PATH . This tells me that it is not the windows_CI being downloaded.

Now I downloaded the zip file format, I get the TensorRT windows_CI, when i run setup.py, i get an error, that this is not a git repository. I ran the command git init, that error is gone, but now I receive this error:

fatal: Needed a single revision Traceback (most recent call last): File "D:\pyth\TensorRT-windows_CI\setup.py", line 104, in <module> __version__ = f"{get_base_version()}.dev0+{get_git_revision_short_hash()}" File "D:\pyth\TensorRT-windows_CI\setup.py", line 43, in get_git_revision_short_hash subprocess.check_output(["git", "rev-parse", "--short", "HEAD"]) File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 421, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['git', 'rev-parse', '--short', 'HEAD']' returned non-zero exit status 128

specifically the line fatal: Needed a single revision I do not understand, and once again, can't find any help online...

Thank you for your help

Hello - thanks for the follow-up. To clone a specific branch from the command line, I use:

git clone --single-branch --branch windows_CI https://github.com/pytorch/TensorRT.git

This should resolve the git issues related to that repository. Then, python setup.py install should avoid bazel.

Hi again @gs-olive, I've successfully installed it like you said, but now when i try to import torch_tensorrt, I get this error:

C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt_Device.py:19: UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. Some dependent features may be unavailable. warnings.warn( Traceback (most recent call last): File "D:\pyth\pytracking-master\band5.py", line 22, in import torch_tensorrt File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt__init.py", line 84, in from torch_tensorrt._compile import * # noqa: F403 File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt_compile.py", line 24, in from torch_tensorrt.dynamo._compiler import compile as dynamo_compile File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\init.py", line 10, in from ._compiler import compile File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo_compiler.py", line 45, in from torch_tensorrt.dynamo.lowering import apply_lowering_passes, get_decompositions File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\lowering\init.py", line 7, in from ._repair_input_aliasing import repair_input_aliasing File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\lowering_repair_input_aliasing.py", line 4, in from torch_tensorrt.dynamo.lowering.passes.pass_utils import get_tensor_placeholders File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\lowering\passes\init__.py", line 1, in from ._aten_lowering_pass import * File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\lowering\passes_aten_lowering_pass.py", line 6, in from .constant_folding import constant_fold File "C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\lowering\passes\constant_folding.py", line 16, in from torch._inductor.constant_folding import ( ModuleNotFoundError: No module named 'torch._inductor.constant_folding'

Thanks for testing that out - I have seen this error before; it relates to the torch version being used. The branch is based on the latest Torch nightly, which has the torch._inductor.constant_folding module, but Torch 2.1.2 does not have that module. I just updated the branch to fix that issue, so windows_CI should get past that issue now if you pull the latest changes and reinstall.

Thank you @gs-olive for your fast reply. Now I'm really confused. I installed torch_tensorrt, now when I try to import torch_tensorrt I get this error and the program stops: C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\_Device.py:19: UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. Some dependent features may be unavailable. warnings.warn(

for some reason at first when I installed it I was having some problems, so I did pip uninstall torch_tensorrt, and then ran again python setup.py install, and I get the error

Or for it to work ok, I need to download the pytorch 2.2.0 nightly?

@gs-olive Hello, I cloned it again as well and it has the same error with "returned non-zero exit status 128" in the end. Can we have some kind of a step by step guide, what exactly we have to do?

1.) Download this that and that. 2.) Add this that and that to the PATH 3.) Then install Python 3.10 4.) Then run setup.py

Something like this I mean.

Thanks a lot.

I still get the following error using the latest commit (https://github.com/pytorch/TensorRT/commit/3c099ef24f8663e578a61bd871a290368e8f37aa at the time of writing) from the windows_CI branch when trying to import torch_tensorrt.

Warning (from warnings module):
  File "C:\Python311\Lib\site-packages\torch_tensorrt\_Device.py", line 19
    warnings.warn(
UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. Some dependent features may be unavailable.
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    import torch_tensorrt
  File "C:\Python311\Lib\site-packages\torch_tensorrt\__init__.py", line 90, in <module>
    from torch_tensorrt.logging import *
  File "C:\Python311\Lib\site-packages\torch_tensorrt\logging.py", line 4, in <module>
    from torch_tensorrt._C import (
ModuleNotFoundError: No module named 'torch_tensorrt._C'

@HolyWu I don't know exactly why the problem of you is happening, because I'm not the dev of this, but perhaps you could try Python 3.10 because the dev said it.

@gs-olive Hello, I cloned it again as well and it has the same error with "returned non-zero exit status 128" in the end. Can we have some kind of a step by step guide, what exactly we have to do?

1.) Download this that and that. 2.) Add this that and that to the PATH 3.) Then install Python 3.10 4.) Then run setup.py

Something like this I mean.

Thanks a lot.

By the way I have solved it myself. Here is how:

1.) Before all of this, please install Python 3.11. 2.) And then do this: "pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121" 3.) Download Cuda 12.1 (only this) 4.) Download Cudnn 5.) Download Nvida Tensorrt 6.) Install Cuda 7.) Copy the cudnn content(whats in it) to the folder C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1. 8.) Put Nvidia Tensorrt to the Enviroment Path. 9.) Clone the Pytorch/Tensorrt via this command ONLY (nothing else does not work): git clone --single-branch --branch windows_CI https://github.com/pytorch/TensorRT.git 10.) Then you have to do: "cd folderwhereithasbeencopied" 11.) Then you have to do: "python setup.py"

Setup should install the library.

However I have not tried more than this for now. Some kind of example would be helpful?

Hello - thanks for the comments on this thread. To address them:

@ninono12345 - the UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. is expected currently, since the version of Torch-TensorRT in the windows_CI branch does not support TorchScript; it only supports the ir="dynamo" and ir="torch_compile" IRs when using this patch: https://github.com/pytorch/pytorch/pull/111313.

@HolyWu - thanks for this catch/report - torch_tensorrt._C is not installed in the Windows build on windows_CI, so that import should not be present in this experimental branch. I've added a commit to remove it.

@jensdraht1999 - the steps provided are very close to what I am using on this experimental branch to get a working build. For a detailed set of steps I am currently using with this experimental branch, here is what I have:

Steps for Dynamo-Only, Python-Runtime-Only Windows build using `windows_CI` Branch

Ensure CUDA is Installed (preferably 12.x). See official CUDA installation steps for Windows if not.
Install your preferred TensorRT Python version. Currently, I use pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com tensorrt==9.2.0.post12.dev5. You can also install other TensorRT versions with the Python bindings using the official TensorRT Windows installation docs, which currently do not use pip.
Install the nightly distribution of Torch, for instance: pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu121
Apply this patch: https://github.com/pytorch/pytorch/pull/111313. Since Torch does not officially support Windows for Dynamo, but most of the required functionality is there, we use this patch as an experimental workaround while the pending issues for torch.compile on Windows are addressed. I currently apply the patch by modifying the torch/_dynamo/eval_frame.py file, for which the location can be found by running pip show torch.
Clone the branch: git clone --single-branch --branch windows_CI https://github.com/pytorch/TensorRT.git
cd into the directory containing the cloned repo. For instance, cd TensorRT
Run python setup.py install to install torch_tensorrt
Open a Python interpreter and run import torch; import torch_tensorrt. There may be warnings about TorchScript and Torchvision, but the import should succeed.
Try an example run, for instance python examples/dynamo/torch_compile_resnet_example.py, which runs ResNet using this tutorial.

Note: Windows support for the Dynamo paths is in an experimental state, and there are no guarantees on functionality + it may not work in some compilation cases since it is under active development. Note: To run torch_tensorrt.compile on Windows using the windows_CI branch, it is required to use ir="dynamo" or ir="torch_compile" and use_python_runtime=True.

@gs-olive I have done all of this. But Step 8 is a problem. Step 9 doesn't really work I think. Please look at the pictures. Would you recommend me to use Python 3.10? By the way I have totally reinstalled Python 3.11 deleted all the sitepackage folder and more to have it all clean and also cuda 12.1.

Step 8:

Step 9: If I run the example file without changing anything:

log.txt

What could be the problem?

Hi - thanks for the follow-up. I do not think Python 3.10 vs 3.11 should make a difference here, though I am using Python 3.10. Regarding the attached image, in the IDE on the left, it seems that there is an error finding a cudnn file, whereas in the IDE on the right, there is just the expected warning. Did anything change between the two runs to make the import succeed, as it does on the left?

Regarding the error logs, that is due to the version of the example which I had linked in the original message. I have fixed the link now, and it uses the correct configuration parameters for Windows. It is linked here as well for convenience.

@gs-olive When I "import torch; import torch_tensorrt" in IDLE, then it will throw multiple .dll "procedure entry point not found" (german "prozedureinsprungpunkt nicht gefunden") errors. After I click them away, then it will run the right one.

The files are actually there, where they are suppossed to be. Please look:

Also the eval file is there where it belongs:

This is the log for example file after that you have linked right now:

d.txt

@gs-olive I tried Python 3.10 as well it did not work. Could you upload your site-packages folder to the internet so I could try it out?

Got it - thanks for sharing this. It seems that the issue there is in Dynamo and not Torch-TensorRT. If the below snippet also fails on your machine, that would indicate a Torch issue.

import torch
import torchvision.models as models

model = models.resnet18(pretrained=True).half().eval().to("cuda")
inputs = [torch.randn((1, 3, 224, 224)).to("cuda").half()]
optimized_model = torch.compile(model, backend="eager")
optimized_model (*inputs)

Additionally, I am not using IDLE as my Python interpreter - I have installed Python for Windows from the Python site directly and I am invoking it in Powershell - I think this could also potentially affect Torch. The versions of the relevant packages on my machine are:

$ python -m pip list
Package                  Version
------------------------ ------------------------ 
tensorrt                 9.2.0.post12.dev5
tensorrt-bindings        9.2.0.post12.dev5
tensorrt-libs            9.2.0.post12.dev5
torch                    2.3.0.dev20240116+cu121

The example you have provided did give me the same error, so my Pytorch Nightly does somehow not work. So I have to work it out.

My depedencies:

C:\Users\King>python -m pip list Package Version

black 23.12.1 certifi 2023.11.17 cfgv 3.4.0 charset-normalizer 3.3.2 clang-format 14.0.6 click 8.1.7 colorama 0.4.6 distlib 0.3.8 exceptiongroup 1.2.0 expecttest 0.2.1 filelock 3.9.0 fsspec 2023.4.0 huggingface-hub 0.20.2 identify 2.5.33 idna 3.6 iniconfig 2.0.0 isort 5.13.2 Jinja2 3.1.2 MarkupSafe 2.1.3 mpmath 1.2.1 mypy 1.8.0 mypy-extensions 1.0.0 networkx 3.0rc1 nodeenv 1.8.0 numpy 1.26.3 nvidia-cublas-cu12 12.3.4.1 nvidia-cuda-nvrtc-cu12 12.3.107 nvidia-cuda-runtime-cu12 12.3.101 nvidia-cudnn-cu12 8.9.7.29 packaging 23.2 parameterized 0.9.0 pathspec 0.12.1 pillow 10.2.0 pip 23.3.2 platformdirs 4.1.0 pluggy 1.3.0 pre-commit 3.6.0 pytest 7.4.4 PyYAML 6.0.1 regex 2023.12.25 requests 2.31.0 ruff 0.1.13 safetensors 0.4.1 setuptools 69.0.3 sympy 1.11.1 tensorrt 9.2.0.post12.dev5 tensorrt-bindings 9.2.0.post12.dev5 tensorrt-libs 9.2.0.post12.dev5 timm 0.9.12 tokenizers 0.15.0 tomli 2.0.1 torch 2.3.0.dev20240117+cu121 torch_tensorrt 2.2.0.dev0+2c304e4b torchvision 0.16.2 tqdm 4.66.1 transformers 4.36.2 typing_extensions 4.8.0 urllib3 2.1.0 virtualenv 20.25.0 wheel 0.42.0

@gs-olive Just one question. I think I have ruled out the problem. I think Torchvision is definitly needed and also installed the nightly version, but this is making problem.

Can i just know from you, which torchvision and torchaudio you have installed. I think this would make this problem go away.

Okay fixed the problem with Torchvision like this. Please try to download them like this:

pip install --pre torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121

While downloading it, you will the see the download link for .whl files. Put the link in a browser, download them. After that put them in a folder in the desktop for example and install via this command:

pip install "C:\Users{UserName}\Desktop\whl files\torchaudio-2.2.0.dev20240118+cu121-cp310-cp310-win_amd64.whl" --force-reinstall --no-deps

pip install "C:\Users{UserName}\Desktop\whl files\torchvision-0.18.0.dev20240118+cu121-cp310-cp310-win_amd64.whl" --force-reinstall --no-deps

This will install those both packages regardless of their dependencies.

However the problem on the original example exists, but I will look at it now.

Thanks for the update. I do not have torchaudio installed, and my versions of torch and torchvision are:

torch                    2.3.0.dev20240116+cu121
torchvision              0.18.0.dev20240116+cu121

I installed these using something very similar to what you have written:

pip uninstall -y torch torchvision
pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu121

@gs-olive Okay now I have fixed it, the rest of the errors are the expected ones. The problem was this:

You have to delete following folder:

"C:\Users{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\nvidia\cudnn"

Because it seems that torch already has those files somehow and now the dll errors disappeard.

Tips: Do NOT USE IDLE. There is really something broke with it. Use Cmd or Powershell or something else.

It really worked with the example, if run through cmd.

There's still an error, but it is definitly loading everything into VRAM.

This error here is excepted, I mean in the end of the script ? @gs-olive

Thanks again for your help and patience. @gs-olive

log.log

Updated with 12.3 cuda and made all the steps again. Now it seems to work better, but I'm not sure, if the example provided is suppossed to have exactly this error here below. And also just deleted the .dll files in the folder "C:\Users{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\nvidia\cudnn", everything else stayed in there. The error seems to be less problematic. And also run in CMD, NOT in IDLE.

log.log

Thanks for the details - the error at the end is not expected and I want to determine whether the issue is in torch.compile or tensorrt. Does an error appear when compiling the Torch model from this comment: https://github.com/pytorch/TensorRT/issues/2577#issuecomment-1897730589?

@gs-olive I do it like this:

The code I have saved in a file called 1.py. The content:

import torch
import torchvision
model = models.resnet18(pretrained=True).half().eval().to("cuda")
inputs = [torch.randn((1, 3, 224, 224)).to("cuda").half()]
optimized_model = torch_tensorrt.compile(model, backend="eager")
optimized_model (*inputs)

I run via CMD. This happens:

C:\Users\{UserName}>C:\Users\{UserName}\Desktop\1.py
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\utils\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
Traceback (most recent call last):
  File "C:\Users\{UserName}\Desktop\1.py", line 5, in <module>
    model = models.resnet18(pretrained=True).half().eval().to("cuda")
NameError: name 'models' is not defined

C:\Users\{UserName}>

But if I run this code here via CMD:

import torch
import torchvision.models as models
model = models.resnet18(pretrained=True).half().eval().to("cuda")
inputs = [torch.randn((1, 3, 224, 224)).to("cuda").half()]
optimized_model = torch_tensorrt.compile(model, backend="eager")
optimized_model (*inputs)

This will happen:

Microsoft Windows [Version 10.0.19045.3930]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

C:\Users\{UserName}>C:\Users\{UserName}\Desktop\whl\1.py
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\utils\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet18_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet18_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Traceback (most recent call last):
  File "C:\Users\{UserName}\Desktop\whl\1.py", line 6, in <module>
    optimized_model = torch_tensorrt.compile(model, backend="eager")
NameError: name 'torch_tensorrt' is not defined

C:\Users\{UserName}>

If I run this code via CMD:

import torch
import torchvision.models as models
import torch_tensorrt
model = models.resnet18(pretrained=True).half().eval().to("cuda")
inputs = [torch.randn((1, 3, 224, 224)).to("cuda").half()]
optimized_model = torch_tensorrt.compile(model, backend="eager")
optimized_model (*inputs)

This happens:

Microsoft Windows [Version 10.0.19045.3930]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

C:\Users\{UserName}>C:\Users\{UserName}\Desktop\whl\1.py
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\utils\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\_Device.py:19: UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. Some dependent features may be unavailable.
  warnings.warn(
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet18_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet18_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
INFO:torch_tensorrt._compile:ir was set to default, using dynamo as ir
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\_dynamo\eval_frame.py:1329: UserWarning: export(f, *args, **kwargs) is deprecated, use export(f)(*args, **kwargs) instead.  If you don't migrate, we may break your export call in the future if your user defined kwargs conflict with future kwargs added to export(f).
  warnings.warn(
Traceback (most recent call last):
  File "C:\Users\{UserName}\Desktop\whl\1.py", line 7, in <module>
    optimized_model = torch_tensorrt.compile(model, backend="eager")
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\_compile.py", line 226, in compile
    exp_program = torch_tensorrt.dynamo.trace(module, torchtrt_inputs, **kwargs)
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\dynamo\_tracer.py", line 85, in trace
    exp_program = export(mod, tuple(torch_inputs), dynamic_shapes=dynamic_shapes)
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\export\__init__.py", line 193, in export
    return _export(
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\export\exported_program.py", line 79, in wrapper
    return fn(*args, **kwargs)
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\export\_trace.py", line 651, in _export
    gm_torch_level = _export_to_torch_ir(
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\export\_trace.py", line 273, in _export_to_torch_ir
    gm_torch_level, _ = torch._dynamo.export(
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\_dynamo\eval_frame.py", line 1334, in export
    return inner(*extra_args, **extra_kwargs)
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\_dynamo\eval_frame.py", line 1220, in inner
    result_traced = opt_f(*args, **kwargs)
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\_dynamo\eval_frame.py", line 410, in _fn
    return fn(*args, **kwargs)
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: ResNet.forward() got an unexpected keyword argument 'disable_constraint_solver'

C:\Users\{UserName}>

I'm not sure, if something I have forgotten somehow. I have followed all your steps, step by step. Here are the pacakages:

The packages:
black                    23.12.1
certifi                  2022.12.7
cfgv                     3.4.0
charset-normalizer       2.1.1
clang-format             14.0.6
click                    8.1.7
colorama                 0.4.6
distlib                  0.3.8
exceptiongroup           1.2.0
expecttest               0.1.6
filelock                 3.13.1
fsspec                   2023.12.2
huggingface-hub          0.20.2
identify                 2.5.33
idna                     3.4
iniconfig                2.0.0
isort                    5.13.2
Jinja2                   3.1.2
MarkupSafe               2.1.3
mpmath                   1.2.1
mypy                     1.8.0
mypy-extensions          1.0.0
networkx                 3.0rc1
nodeenv                  1.8.0
numpy                    1.24.1
nvidia-cublas-cu12       12.3.4.1
nvidia-cuda-nvrtc-cu12   12.3.107
nvidia-cuda-runtime-cu12 12.3.101
nvidia-cudnn-cu12        8.9.7.29
packaging                23.2
parameterized            0.9.0
pathspec                 0.12.1
Pillow                   9.3.0
pip                      23.3.2
platformdirs             4.1.0
pluggy                   1.3.0
pre-commit               3.6.0
pytest                   7.4.4
PyYAML                   6.0.1
regex                    2023.12.25
requests                 2.28.1
ruff                     0.1.14
safetensors              0.4.1
setuptools               65.5.0
sympy                    1.11.1
tensorrt                 9.2.0.post12.dev5
tensorrt-bindings        9.2.0.post12.dev5
tensorrt-libs            9.2.0.post12.dev5
timm                     0.9.12
tokenizers               0.15.0
tomli                    2.0.1
torch                    2.3.0.dev20240120+cu121
torch-tensorrt           2.3.0.dev0+e7e291144
torchvision              0.18.0.dev20240120+cu121
tqdm                     4.66.1
transformers             4.36.2
typing_extensions        4.8.0
urllib3                  1.26.13
virtualenv               20.25.0
wheel                    0.42.0

If I clone to and then run: python setup.py install via CMD, this is what I get:

Microsoft Windows [Version 10.0.19045.3930]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

C:\Users\{UserName}>cd C:\Users\{UserName}\source\repos\TensorRT

C:\Users\{UserName}\source\repos\TensorRT>python setup.py install
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\config\pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  warnings.warn(msg, _BetaConfiguration)
running install
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
creating version file
running build
running build_py
copying py\torch_tensorrt\_version.py -> build\lib\torch_tensorrt
running install_lib
copying build\lib\torch_tensorrt\_version.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\_version.py to _version.cpython-310.pyc
running install_egg_info
running egg_info
writing py\torch_tensorrt.egg-info\PKG-INFO
writing dependency_links to py\torch_tensorrt.egg-info\dependency_links.txt
writing requirements to py\torch_tensorrt.egg-info\requires.txt
writing top-level names to py\torch_tensorrt.egg-info\top_level.txt
reading manifest file 'py\torch_tensorrt.egg-info\SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'py\torch_tensorrt.egg-info\SOURCES.txt'
removing 'C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt-2.3.0.dev0+e7e291144-py3.10.egg-info' (and everything under it)
Copying py\torch_tensorrt.egg-info to C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt-2.3.0.dev0+e7e291144-py3.10.egg-info
running install_scripts

C:\Users\{UserName}\source\repos\TensorRT>

Also if delete the folder in the site-packages folder torch_tensorrt and torch_tensorrt-2.3.0.dev0+e7e92911-py3.10.egg-info and run the code again. This happens and the folder are created there again:

Microsoft Windows [Version 10.0.19045.3930]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

C:\Users\{UserName}>cd C:\Users\{UserName}\source\repos\TensorRT

C:\Users\{UserName}\source\repos\TensorRT>python setup.py install
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\config\pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  warnings.warn(msg, _BetaConfiguration)
running install
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
creating version file
running build
running build_py
copying py\torch_tensorrt\_version.py -> build\lib\torch_tensorrt
running install_lib
copying build\lib\torch_tensorrt\_version.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\_version.py to _version.cpython-310.pyc
running install_egg_info
running egg_info
writing py\torch_tensorrt.egg-info\PKG-INFO
writing dependency_links to py\torch_tensorrt.egg-info\dependency_links.txt
writing requirements to py\torch_tensorrt.egg-info\requires.txt
writing top-level names to py\torch_tensorrt.egg-info\top_level.txt
reading manifest file 'py\torch_tensorrt.egg-info\SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'py\torch_tensorrt.egg-info\SOURCES.txt'
removing 'C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt-2.3.0.dev0+e7e291144-py3.10.egg-info' (and everything under it)
Copying py\torch_tensorrt.egg-info to C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt-2.3.0.dev0+e7e291144-py3.10.egg-info
running install_scripts

C:\Users\{UserName}\source\repos\TensorRT>python setup.py install
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\config\pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  warnings.warn(msg, _BetaConfiguration)
running install
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
creating version file
running build
running build_py
copying py\torch_tensorrt\_version.py -> build\lib\torch_tensorrt
running install_lib
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\backend
copying build\lib\torch_tensorrt\dynamo\backend\backends.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\backend
copying build\lib\torch_tensorrt\dynamo\backend\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\backend
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
copying build\lib\torch_tensorrt\dynamo\conversion\aten_ops_converters.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
copying build\lib\torch_tensorrt\dynamo\conversion\converter_utils.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\activation
copying build\lib\torch_tensorrt\dynamo\conversion\impl\activation\base.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\activation
copying build\lib\torch_tensorrt\dynamo\conversion\impl\activation\ops.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\activation
copying build\lib\torch_tensorrt\dynamo\conversion\impl\activation\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\activation
copying build\lib\torch_tensorrt\dynamo\conversion\impl\addmm.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\attention.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\cast.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\cat.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\condition
copying build\lib\torch_tensorrt\dynamo\conversion\impl\condition\ops.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\condition
copying build\lib\torch_tensorrt\dynamo\conversion\impl\condition\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\condition
copying build\lib\torch_tensorrt\dynamo\conversion\impl\conv.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\deconv.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\elementwise
copying build\lib\torch_tensorrt\dynamo\conversion\impl\elementwise\base.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\elementwise
copying build\lib\torch_tensorrt\dynamo\conversion\impl\elementwise\ops.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\elementwise
copying build\lib\torch_tensorrt\dynamo\conversion\impl\elementwise\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\elementwise
copying build\lib\torch_tensorrt\dynamo\conversion\impl\embedding.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\grid.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\linear.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\matmul.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\normalization
copying build\lib\torch_tensorrt\dynamo\conversion\impl\normalization\ops.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\normalization
copying build\lib\torch_tensorrt\dynamo\conversion\impl\normalization\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\normalization
copying build\lib\torch_tensorrt\dynamo\conversion\impl\pad.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\permutation.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\pool.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\reduce.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\select.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\shape.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\shuffle.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\slice
copying build\lib\torch_tensorrt\dynamo\conversion\impl\slice\base.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\slice
copying build\lib\torch_tensorrt\dynamo\conversion\impl\slice\ops.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\slice
copying build\lib\torch_tensorrt\dynamo\conversion\impl\slice\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\slice
copying build\lib\torch_tensorrt\dynamo\conversion\impl\split.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\squeeze.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\topk.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\unary
copying build\lib\torch_tensorrt\dynamo\conversion\impl\unary\base.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\unary
copying build\lib\torch_tensorrt\dynamo\conversion\impl\unary\ops.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\unary
copying build\lib\torch_tensorrt\dynamo\conversion\impl\unary\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\unary
copying build\lib\torch_tensorrt\dynamo\conversion\impl\unsqueeze.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\upsample.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\impl\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl
copying build\lib\torch_tensorrt\dynamo\conversion\ops_evaluators.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
copying build\lib\torch_tensorrt\dynamo\conversion\prims_ops_converters.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
copying build\lib\torch_tensorrt\dynamo\conversion\truncate_long_and_double.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
copying build\lib\torch_tensorrt\dynamo\conversion\_conversion.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
copying build\lib\torch_tensorrt\dynamo\conversion\_ConversionContext.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
copying build\lib\torch_tensorrt\dynamo\conversion\_ConverterRegistry.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
copying build\lib\torch_tensorrt\dynamo\conversion\_TRTInterpreter.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
copying build\lib\torch_tensorrt\dynamo\conversion\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\constant_folding.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\fuse_prims_broadcast.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\lower_linear.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\lower_scaled_dot_product_attention.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\pass_manager.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\pass_utils.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\remove_input_alias_fixing_clones.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\repair_input_as_output.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\replace_max_pool_with_indices.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\view_to_reshape.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\_aten_lowering_pass.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\passes\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes
copying build\lib\torch_tensorrt\dynamo\lowering\_decompositions.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering
copying build\lib\torch_tensorrt\dynamo\lowering\_decomposition_groups.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering
copying build\lib\torch_tensorrt\dynamo\lowering\_fusers.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering
copying build\lib\torch_tensorrt\dynamo\lowering\_repair_input_aliasing.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering
copying build\lib\torch_tensorrt\dynamo\lowering\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\partitioning
copying build\lib\torch_tensorrt\dynamo\partitioning\common.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\partitioning
copying build\lib\torch_tensorrt\dynamo\partitioning\_adjacency_partitioner.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\partitioning
copying build\lib\torch_tensorrt\dynamo\partitioning\_global_partitioner.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\partitioning
copying build\lib\torch_tensorrt\dynamo\partitioning\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\partitioning
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\runtime
copying build\lib\torch_tensorrt\dynamo\runtime\tools.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\runtime
copying build\lib\torch_tensorrt\dynamo\runtime\_PythonTorchTensorRTModule.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\runtime
copying build\lib\torch_tensorrt\dynamo\runtime\_TorchTensorRTModule.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\runtime
copying build\lib\torch_tensorrt\dynamo\runtime\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\runtime
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\tools
copying build\lib\torch_tensorrt\dynamo\tools\opset_coverage.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\tools
copying build\lib\torch_tensorrt\dynamo\tools\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\tools
copying build\lib\torch_tensorrt\dynamo\utils.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo
copying build\lib\torch_tensorrt\dynamo\_compiler.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo
copying build\lib\torch_tensorrt\dynamo\_defaults.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo
copying build\lib\torch_tensorrt\dynamo\_DryRunTracker.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo
copying build\lib\torch_tensorrt\dynamo\_exporter.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo
copying build\lib\torch_tensorrt\dynamo\_settings.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo
copying build\lib\torch_tensorrt\dynamo\_SourceIR.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo
copying build\lib\torch_tensorrt\dynamo\_tracer.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo
copying build\lib\torch_tensorrt\dynamo\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\acc_ops_converters.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\adaptive_avgpool.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\add.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\aten_ops_converters.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\batchnorm.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\converter_utils.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\impl
copying build\lib\torch_tensorrt\fx\converters\impl\activation.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\impl
copying build\lib\torch_tensorrt\fx\converters\impl\convolution.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\impl
copying build\lib\torch_tensorrt\fx\converters\impl\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\impl
copying build\lib\torch_tensorrt\fx\converters\linear.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\maxpool.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\mul.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\nn_ops_converters.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\quantization.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\transformation.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converters\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters
copying build\lib\torch_tensorrt\fx\converter_registry.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
copying build\lib\torch_tensorrt\fx\diagnostics.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
copying build\lib\torch_tensorrt\fx\fx2trt.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
copying build\lib\torch_tensorrt\fx\input_tensor_spec.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
copying build\lib\torch_tensorrt\fx\lower.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
copying build\lib\torch_tensorrt\fx\lower_setting.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
copying build\lib\torch_tensorrt\fx\observer.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes
copying build\lib\torch_tensorrt\fx\passes\graph_opts.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes
copying build\lib\torch_tensorrt\fx\passes\lower_basic_pass.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes
copying build\lib\torch_tensorrt\fx\passes\lower_basic_pass_aten.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes
copying build\lib\torch_tensorrt\fx\passes\lower_pass_manager_builder.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes
copying build\lib\torch_tensorrt\fx\passes\pass_utils.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes
copying build\lib\torch_tensorrt\fx\passes\remove_duplicate_output_args.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes
copying build\lib\torch_tensorrt\fx\passes\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\common_fx2trt.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\engine_layer_visualize.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\graph_util.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\model_packager.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\node_profiler.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\tensor_prop.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\timing_cache_utils.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\trt_minimizer.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\trt_profiler_sorted.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\trt_splitter.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
copying build\lib\torch_tensorrt\fx\tools\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer
copying build\lib\torch_tensorrt\fx\tracer\acc_tracer\acc_normalizer.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer
copying build\lib\torch_tensorrt\fx\tracer\acc_tracer\acc_ops.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer
copying build\lib\torch_tensorrt\fx\tracer\acc_tracer\acc_op_properties.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer
copying build\lib\torch_tensorrt\fx\tracer\acc_tracer\acc_shape_prop.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer
copying build\lib\torch_tensorrt\fx\tracer\acc_tracer\acc_tracer.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer
copying build\lib\torch_tensorrt\fx\tracer\acc_tracer\acc_utils.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer
copying build\lib\torch_tensorrt\fx\tracer\acc_tracer\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\dispatch_tracer
copying build\lib\torch_tensorrt\fx\tracer\dispatch_tracer\aten_tracer.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\dispatch_tracer
copying build\lib\torch_tensorrt\fx\tracer\dispatch_tracer\tracer.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\dispatch_tracer
copying build\lib\torch_tensorrt\fx\tracer\dispatch_tracer\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\dispatch_tracer
copying build\lib\torch_tensorrt\fx\tracer\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer
copying build\lib\torch_tensorrt\fx\trt_module.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
copying build\lib\torch_tensorrt\fx\types.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
copying build\lib\torch_tensorrt\fx\utils.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
copying build\lib\torch_tensorrt\fx\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx
creating C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\runtime
copying build\lib\torch_tensorrt\runtime\multi_device_safe_mode.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\runtime
copying build\lib\torch_tensorrt\runtime\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\runtime
copying build\lib\torch_tensorrt\_compile.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt
copying build\lib\torch_tensorrt\_Device.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt
copying build\lib\torch_tensorrt\_enums.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt
copying build\lib\torch_tensorrt\_Input.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt
copying build\lib\torch_tensorrt\_utils.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt
copying build\lib\torch_tensorrt\_version.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt
copying build\lib\torch_tensorrt\__init__.py -> C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\backend\backends.py to backends.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\backend\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\aten_ops_converters.py to aten_ops_converters.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\converter_utils.py to converter_utils.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\activation\base.py to base.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\activation\ops.py to ops.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\activation\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\addmm.py to addmm.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\attention.py to attention.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\cast.py to cast.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\cat.py to cat.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\condition\ops.py to ops.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\condition\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\conv.py to conv.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\deconv.py to deconv.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\elementwise\base.py to base.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\elementwise\ops.py to ops.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\elementwise\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\embedding.py to embedding.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\grid.py to grid.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\linear.py to linear.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\matmul.py to matmul.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\normalization\ops.py to ops.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\normalization\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\pad.py to pad.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\permutation.py to permutation.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\pool.py to pool.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\reduce.py to reduce.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\select.py to select.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\shape.py to shape.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\shuffle.py to shuffle.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\slice\base.py to base.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\slice\ops.py to ops.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\slice\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\split.py to split.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\squeeze.py to squeeze.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\topk.py to topk.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\unary\base.py to base.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\unary\ops.py to ops.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\unary\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\unsqueeze.py to unsqueeze.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\upsample.py to upsample.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\ops_evaluators.py to ops_evaluators.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\prims_ops_converters.py to prims_ops_converters.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\truncate_long_and_double.py to truncate_long_and_double.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\_conversion.py to _conversion.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\_ConversionContext.py to _ConversionContext.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\_ConverterRegistry.py to _ConverterRegistry.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\_TRTInterpreter.py to _TRTInterpreter.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\conversion\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\constant_folding.py to constant_folding.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\fuse_prims_broadcast.py to fuse_prims_broadcast.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\lower_linear.py to lower_linear.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\lower_scaled_dot_product_attention.py to lower_scaled_dot_product_attention.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\pass_manager.py to pass_manager.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\pass_utils.py to pass_utils.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\remove_input_alias_fixing_clones.py to remove_input_alias_fixing_clones.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\repair_input_as_output.py to repair_input_as_output.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\replace_max_pool_with_indices.py to replace_max_pool_with_indices.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\view_to_reshape.py to view_to_reshape.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\_aten_lowering_pass.py to _aten_lowering_pass.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\passes\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\_decompositions.py to _decompositions.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\_decomposition_groups.py to _decomposition_groups.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\_fusers.py to _fusers.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\_repair_input_aliasing.py to _repair_input_aliasing.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\lowering\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\partitioning\common.py to common.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\partitioning\_adjacency_partitioner.py to _adjacency_partitioner.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\partitioning\_global_partitioner.py to _global_partitioner.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\partitioning\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\runtime\tools.py to tools.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\runtime\_PythonTorchTensorRTModule.py to _PythonTorchTensorRTModule.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\runtime\_TorchTensorRTModule.py to _TorchTensorRTModule.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\runtime\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\tools\opset_coverage.py to opset_coverage.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\tools\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\utils.py to utils.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\_compiler.py to _compiler.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\_defaults.py to _defaults.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\_DryRunTracker.py to _DryRunTracker.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\_exporter.py to _exporter.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\_settings.py to _settings.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\_SourceIR.py to _SourceIR.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\_tracer.py to _tracer.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\dynamo\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\acc_ops_converters.py to acc_ops_converters.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\adaptive_avgpool.py to adaptive_avgpool.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\add.py to add.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\aten_ops_converters.py to aten_ops_converters.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\batchnorm.py to batchnorm.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\converter_utils.py to converter_utils.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\impl\activation.py to activation.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\impl\convolution.py to convolution.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\impl\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\linear.py to linear.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\maxpool.py to maxpool.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\mul.py to mul.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\nn_ops_converters.py to nn_ops_converters.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\quantization.py to quantization.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\transformation.py to transformation.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converters\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\converter_registry.py to converter_registry.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\diagnostics.py to diagnostics.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\fx2trt.py to fx2trt.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\input_tensor_spec.py to input_tensor_spec.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\lower.py to lower.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\lower_setting.py to lower_setting.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\observer.py to observer.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes\graph_opts.py to graph_opts.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes\lower_basic_pass.py to lower_basic_pass.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes\lower_basic_pass_aten.py to lower_basic_pass_aten.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes\lower_pass_manager_builder.py to lower_pass_manager_builder.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes\pass_utils.py to pass_utils.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes\remove_duplicate_output_args.py to remove_duplicate_output_args.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\passes\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\common_fx2trt.py to common_fx2trt.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\engine_layer_visualize.py to engine_layer_visualize.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\graph_util.py to graph_util.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\model_packager.py to model_packager.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\node_profiler.py to node_profiler.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\tensor_prop.py to tensor_prop.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\timing_cache_utils.py to timing_cache_utils.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\trt_minimizer.py to trt_minimizer.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\trt_profiler_sorted.py to trt_profiler_sorted.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\trt_splitter.py to trt_splitter.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tools\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer\acc_normalizer.py to acc_normalizer.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer\acc_ops.py to acc_ops.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer\acc_op_properties.py to acc_op_properties.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer\acc_shape_prop.py to acc_shape_prop.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer\acc_tracer.py to acc_tracer.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer\acc_utils.py to acc_utils.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\acc_tracer\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\dispatch_tracer\aten_tracer.py to aten_tracer.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\dispatch_tracer\tracer.py to tracer.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\dispatch_tracer\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\tracer\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\trt_module.py to trt_module.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\types.py to types.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\utils.py to utils.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\fx\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\runtime\multi_device_safe_mode.py to multi_device_safe_mode.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\runtime\__init__.py to __init__.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\_compile.py to _compile.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\_Device.py to _Device.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\_enums.py to _enums.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\_Input.py to _Input.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\_utils.py to _utils.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\_version.py to _version.cpython-310.pyc
byte-compiling C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt\__init__.py to __init__.cpython-310.pyc
running install_egg_info
running egg_info
writing py\torch_tensorrt.egg-info\PKG-INFO
writing dependency_links to py\torch_tensorrt.egg-info\dependency_links.txt
writing requirements to py\torch_tensorrt.egg-info\requires.txt
writing top-level names to py\torch_tensorrt.egg-info\top_level.txt
reading manifest file 'py\torch_tensorrt.egg-info\SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'py\torch_tensorrt.egg-info\SOURCES.txt'
Copying py\torch_tensorrt.egg-info to C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_tensorrt-2.3.0.dev0+e7e291144-py3.10.egg-info
running install_scripts

C:\Users\{UserName}\source\repos\TensorRT>

So what do we now?

This will happen in Powershell if import is done:

PS C:\Users{UserName}> python Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

import torch; import torch_tensorrt C:\Users{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\utils\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( C:\Users{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt_Device.py:19: UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. Some dependent features may be unavailable. warnings.warn(

WAIT A MINUTE:

Could it be, that regular GPUs like RTX 3060 LAPTOP are not supported by Tensorrt at all?

I thought gs-olive had a typo in https://github.com/pytorch/TensorRT/issues/2577#issuecomment-1897730589. It should be optimized_model = torch.compile(model, backend="eager"), not optimized_model = torch_tensorrt.compile(model, backend="eager").

import torch
import torchvision.models as models

model = models.resnet18(pretrained=True).half().eval().to("cuda")
inputs = [torch.randn((1, 3, 224, 224)).to("cuda").half()]
optimized_model = torch.compile(model, backend="eager")
print(optimized_model(*inputs))

It has worked with the code you have provided. Thank you. @HolyWu

Here is the what is the output:

Microsoft Windows [Version 10.0.19045.3930]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

C:\Users\{UserName}>C:\Users\{UserName}\Desktop\whl\2.py
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\utils\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torchvision\models\_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torchvision\models\_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet18_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet18_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
tensor([[ 1.0977e+00,  2.5957e+00,  2.9277e+00,  2.9453e+00,  4.3242e+00,
          3.6836e+00,  3.9395e+00,  3.5767e-01, -8.9941e-01, -6.6797e-01,
         -7.0508e-01,  1.2744e+00,  1.7505e-01,  1.2754e+00,  1.5352e+00,
          7.6611e-01, -7.4854e-01, -2.7759e-01,  1.2705e+00,  3.0103e-01,
         -5.8350e-02, -2.9272e-01,  1.4014e+00,  4.0552e-01, -1.9031e-01,
          2.1399e-01,  8.3545e-01,  6.3867e-01,  3.1055e-01,  7.8662e-01,
          1.3662e+00,  3.8574e-01, -6.9531e-01,  2.6582e+00,  3.2012e+00,
          6.6846e-01,  4.0527e-01,  4.0747e-01,  3.3057e-01,  1.9170e+00,
          1.6221e+00,  6.8115e-01,  9.9951e-01,  4.8120e-01,  1.3926e+00,
          2.6367e-01,  1.7412e+00, -1.1719e+00,  1.6943e+00,  6.7334e-01,
          1.8330e+00, -1.7939e+00,  6.2500e-01,  5.9521e-01, -1.8408e-01,
          1.3535e+00, -8.2666e-01,  3.0469e-01,  1.9014e+00,  2.4866e-01,
          1.4521e+00, -1.1499e-01,  2.1561e-02, -1.1504e+00,  3.0518e-01,
          4.3438e+00,  5.5725e-02, -2.7368e-01,  3.2568e-01,  1.9385e+00,
          1.6504e+00,  2.1035e+00,  5.1904e-01,  8.2959e-01,  6.6504e-01,
          1.9297e+00,  8.6328e-01,  1.4189e+00,  1.3486e+00,  2.4277e+00,
         -1.5957e+00,  1.1562e+00, -2.5269e-01, -2.8730e+00,  5.9424e-01,
          7.2021e-01, -5.2148e-01, -4.9268e-01, -2.8589e-01,  1.5371e+00,
         -1.2080e+00, -1.1064e+00,  5.2246e-01, -4.7583e-01,  3.2788e-01,
         -2.4141e+00,  1.1993e-01, -8.3740e-01, -1.5615e+00,  1.2524e-01,
         -1.2732e-01, -1.9434e+00, -1.5410e+00,  2.0984e-01,  4.2871e-01,
         -2.2095e-01, -1.1926e-01,  6.4531e+00,  2.2832e+00,  2.4609e+00,
          1.8496e+00,  4.0039e+00,  2.6035e+00,  1.8965e+00,  1.9883e+00,
          1.4610e-02,  9.0234e-01,  1.3057e+00,  4.6729e-01,  3.9331e-01,
          5.5713e-01,  1.2622e-01,  6.1035e-01,  1.8857e+00,  2.4961e+00,
          1.8994e+00,  2.5059e+00, -6.2195e-02,  1.3013e-01,  9.8291e-01,
         -2.0239e-01,  2.4390e-01,  1.7998e+00,  3.2251e-01,  5.9473e-01,
         -7.1143e-01, -1.3359e+00,  2.2888e-01,  5.2246e-01, -3.9453e-01,
         -2.1021e-01,  1.4319e-01, -3.4180e-01,  5.2148e-01,  2.3230e-01,
          5.0293e-01, -5.8350e-01,  3.1201e-01,  7.1631e-01,  2.1172e+00,
          1.2432e+00, -2.9224e-01, -2.5527e+00,  4.9756e-01, -7.1582e-01,
         -5.0690e-02, -1.3408e+00, -8.1152e-01, -7.7637e-01, -8.9795e-01,
         -2.5566e+00, -7.3059e-02,  6.5430e-02, -7.1680e-01, -1.0459e+00,
         -1.4453e+00, -1.0342e+00, -2.2910e+00, -1.0391e+00, -1.1230e+00,
         -1.8203e+00, -1.0797e-01, -1.5186e+00, -1.9409e-01, -1.1855e+00,
         -2.6348e+00, -1.3877e+00, -2.3027e+00,  8.3740e-01, -1.0830e+00,
         -5.2783e-01, -8.6816e-01, -9.4727e-01, -1.1318e+00, -1.3525e+00,
         -9.4482e-01, -1.7070e+00, -4.6753e-01, -2.5898e+00, -7.5293e-01,
         -1.2969e+00, -1.4766e+00, -9.5264e-01, -1.8721e+00, -2.3066e+00,
          1.3086e-01, -1.0898e+00, -2.6055e+00, -2.2012e+00, -9.4434e-01,
         -2.7383e+00,  9.8999e-02, -1.0615e+00,  1.2744e+00,  9.3555e-01,
         -1.7930e+00, -1.8076e+00, -2.5928e-01, -1.1924e+00,  1.0046e-01,
         -1.9707e+00, -1.3594e+00, -1.1045e+00, -5.8252e-01, -2.1367e+00,
         -1.1055e+00, -1.1748e+00, -3.3618e-01, -7.0215e-01, -1.3945e+00,
         -2.8984e+00, -1.2021e+00, -1.8428e+00, -2.6929e-01, -5.8594e-01,
         -1.4053e+00, -1.8711e+00, -1.9717e+00, -2.7637e+00, -9.6436e-01,
         -3.6011e-01, -1.1240e+00, -2.0959e-01, -3.0273e+00, -8.7695e-01,
         -1.4375e+00, -8.9404e-01, -1.0234e+00, -8.5693e-01, -1.4072e+00,
         -1.1689e+00, -1.4766e+00, -4.6631e-01, -1.2041e+00, -4.1797e+00,
         -3.8965e-01, -1.8516e+00, -1.2158e+00, -3.7720e-01, -1.5732e+00,
         -6.2354e-01, -5.2490e-01, -3.0254e+00, -5.4688e-01, -5.6982e-01,
         -2.3516e+00, -2.4004e+00, -1.0293e+00, -6.5625e-01,  3.2684e-02,
         -1.9229e+00, -1.6113e+00, -3.4121e+00, -3.7842e-01, -1.2832e+00,
         -5.8887e-01, -1.0410e+00, -2.1445e+00, -2.3770e+00, -1.2520e+00,
         -5.8398e-01, -1.7959e+00,  8.3801e-02, -1.8730e+00, -1.4395e+00,
         -2.3340e+00, -2.3828e+00, -8.3643e-01, -1.4102e+00,  6.6064e-01,
          3.7158e-01,  4.6631e-01,  7.3242e-01, -1.0278e-01,  5.5908e-01,
         -2.0190e-01, -1.1641e+00, -9.8877e-01, -6.8408e-01, -1.3857e+00,
         -2.6191e+00, -1.7529e+00, -8.7305e-01, -1.8965e+00, -1.3340e+00,
         -2.0020e-01,  4.5825e-01, -2.9941e+00,  6.4062e-01, -7.8809e-01,
          1.1432e-01,  1.3848e+00,  6.0107e-01,  1.5820e+00, -6.5247e-02,
          3.6255e-02,  7.3877e-01,  9.7266e-01,  6.1084e-01,  2.1167e-01,
          3.0000e+00,  1.3105e+00,  1.8643e+00,  1.9268e+00,  2.9258e+00,
          2.5234e+00,  1.6104e+00,  2.6523e+00,  3.3184e+00,  1.1943e+00,
          4.8755e-01, -3.2520e-01, -1.2656e+00, -3.0225e-01,  1.6631e+00,
          6.2637e-03,  4.6851e-01,  3.7051e+00,  1.3770e+00,  3.0938e+00,
          1.6760e-01,  3.3667e-01, -2.6538e-01,  7.9346e-02,  3.2275e-01,
          4.3286e-01, -2.3804e-01, -4.3774e-01, -4.1351e-02, -1.9609e+00,
         -1.7285e+00, -1.8770e+00, -1.3252e+00, -7.5000e-01, -1.1774e-01,
         -1.4834e+00, -2.2363e+00, -1.3203e+00, -1.5596e+00, -7.0020e-01,
         -2.2864e-01, -1.7383e+00, -1.1133e+00, -6.5820e-01, -1.6035e+00,
         -2.3613e+00,  3.1494e-01, -3.6426e-01,  2.1240e-01,  5.1575e-02,
         -4.9243e-01,  1.9980e+00,  1.9653e-01,  1.2529e+00, -1.2793e+00,
         -7.5537e-01, -2.3242e+00, -1.5264e+00, -1.3252e+00, -1.1777e+00,
          6.4209e-02, -2.5801e+00, -1.8984e+00, -3.5767e-01,  7.4219e-01,
         -8.0225e-01, -5.4736e-01, -1.6187e-01, -1.2881e+00, -1.1865e+00,
         -8.4473e-01, -7.0850e-01, -4.7021e-01,  2.4707e-01, -1.1416e+00,
         -1.6260e+00, -1.3623e+00, -1.4531e+00, -1.8467e+00,  6.8799e-01,
          2.9473e+00,  2.4746e+00,  1.8848e+00,  1.7402e+00,  1.7656e+00,
          1.9551e+00,  2.7012e+00,  2.4707e+00, -1.0919e-01, -4.3481e-01,
         -1.6240e+00, -1.0742e-01,  1.2666e+00, -4.3408e-01, -6.9092e-01,
          1.1934e+00, -4.7827e-01, -2.2578e+00, -2.1660e+00,  1.6143e+00,
          1.3174e+00, -2.4011e-01,  1.2139e+00,  1.8030e-01, -7.4890e-02,
         -1.9629e+00, -4.4238e-01,  1.0049e+00,  1.2900e+00,  1.9473e+00,
         -1.9470e-02,  1.2637e+00,  7.5928e-01, -2.2715e+00, -1.7344e+00,
          1.0488e+00,  6.3599e-02,  7.5732e-01,  1.4746e-01,  2.2305e+00,
         -4.4849e-01, -2.0020e+00, -2.4768e-01,  9.0088e-01, -6.3037e-01,
         -1.0586e+00, -1.0068e+00,  9.1113e-01,  5.8252e-01, -2.2207e+00,
          2.4727e+00,  2.0898e+00, -6.2256e-01, -6.6406e-02, -8.1885e-01,
         -3.4229e-01,  1.7549e+00,  4.2041e-01,  7.0117e-01, -6.5735e-02,
         -2.4238e+00,  7.2363e-01, -7.5049e-01, -8.7646e-01, -1.2002e+00,
          2.9746e+00,  3.6328e-01, -5.8472e-02,  1.8691e+00,  7.0020e-01,
          1.3408e+00,  1.7249e-01,  8.6182e-01,  2.2246e+00,  2.8735e-01,
         -1.5615e+00, -1.9434e+00, -2.3027e+00,  1.5146e+00, -2.1167e-01,
         -4.1968e-01, -6.9275e-02, -3.0078e-01, -4.2310e-01,  1.7949e+00,
         -6.8604e-01, -3.7285e+00, -1.0791e+00,  5.2881e-01,  6.8652e-01,
          8.1982e-01,  2.4707e+00, -2.0874e-01,  2.0176e+00,  2.8137e-02,
          6.5002e-02, -2.2766e-01, -5.5420e-01,  3.4238e+00,  3.5098e+00,
          1.5889e+00,  1.7273e-01,  5.8301e-01, -2.1074e+00,  8.7207e-01,
         -1.6396e+00, -3.8062e-01,  1.6201e+00,  5.7031e-01, -2.0703e-01,
         -2.6306e-02, -4.5728e-01,  1.1768e+00, -5.1172e-01, -3.3691e-01,
         -1.2317e-01,  2.0156e+00,  1.8145e+00,  1.2148e+00, -1.4834e+00,
         -1.7246e+00, -1.8730e+00,  1.0488e+00, -2.5293e-01,  1.0834e-01,
          3.7628e-02, -1.7812e+00,  4.9316e-01, -4.2017e-01, -4.6411e-01,
         -2.2734e+00, -1.2510e+00,  2.6123e-01,  1.5361e+00, -9.5117e-01,
         -7.5867e-02, -1.6426e+00, -1.2422e+00,  1.0358e-01, -2.5586e-01,
          1.2188e+00, -3.6768e-01, -1.9277e+00,  7.7197e-01, -1.7256e+00,
         -3.4058e-01, -5.8545e-01, -4.1870e-01,  2.1641e+00,  3.0312e+00,
         -7.4902e-01, -1.4038e-01,  1.0039e+00,  5.9766e-01,  7.2998e-01,
          1.0488e+00,  2.4353e-01, -2.4023e+00, -1.8184e+00,  1.8008e+00,
         -3.1973e+00, -1.1387e+00,  6.2158e-01, -4.8535e-01, -1.1104e+00,
         -2.0664e+00, -5.8740e-01,  1.0508e+00,  1.0498e-01,  1.6426e+00,
          1.2988e+00, -1.1484e+00,  3.3594e+00,  1.5801e+00, -1.4863e+00,
         -1.0674e+00,  1.9177e-01,  8.1689e-01, -1.9287e+00, -9.9170e-01,
          6.2305e-01, -9.0967e-01,  3.4155e-01, -9.9121e-01,  2.7715e+00,
         -1.8203e+00, -1.0645e+00,  3.5913e-01, -4.6265e-01, -3.3447e-01,
         -4.3774e-01,  8.0713e-01, -2.2617e+00, -1.0996e+00, -5.7031e-01,
         -1.0752e+00, -1.9199e+00,  1.6055e+00, -7.1045e-01, -7.8125e-02,
         -1.2812e+00, -1.3733e-01,  5.6671e-02,  6.5479e-01,  1.0809e-01,
         -8.9355e-01,  5.8350e-01,  1.3906e+00,  3.3398e-01,  1.4561e+00,
          2.2441e+00, -7.5732e-01, -2.6147e-01, -2.7852e+00,  1.5566e+00,
          1.1104e+00,  2.2285e+00, -1.0664e+00,  2.2969e+00, -1.7822e+00,
          2.1802e-01,  5.1094e+00, -2.1543e+00,  1.5117e+00, -1.4912e+00,
          1.1982e+00,  2.8457e+00, -1.7617e+00,  3.5229e-01,  7.3682e-01,
          8.9160e-01, -2.6196e-01,  2.8223e+00,  8.0859e-01, -9.0430e-01,
         -1.4727e+00,  3.3350e-01, -1.2910e+00, -6.5063e-02, -5.0391e-01,
          1.0199e-01, -2.8540e-01,  1.1738e+00,  1.6318e+00, -2.9199e+00,
          1.4209e-01, -5.0439e-01,  1.3604e+00,  4.0894e-01,  3.4741e-01,
          3.4219e+00, -8.7939e-01, -1.7646e+00,  6.1670e-01,  3.7578e+00,
         -9.2236e-01,  2.0801e+00,  3.2251e-01, -1.3809e+00,  9.1504e-01,
          1.2588e+00, -1.2891e+00, -1.5107e+00,  6.7480e-01, -1.5264e+00,
          7.2949e-01, -1.3252e+00,  1.1143e+00,  9.1357e-01,  2.4487e-01,
         -8.6865e-01, -1.5322e+00,  1.1064e+00,  5.3760e-01,  1.3438e+00,
         -1.1377e-01,  1.1318e+00,  8.1592e-01,  1.0322e+00,  4.2505e-01,
         -1.8721e+00, -2.8732e-02,  6.5723e-01,  1.6133e+00,  2.0840e+00,
         -2.0371e+00, -1.9785e+00,  2.8926e+00, -1.2715e+00,  1.7461e+00,
         -2.3047e-01,  9.5508e-01,  1.7188e+00,  3.3569e-01,  4.0674e-01,
          3.8794e-01, -4.6045e-01, -1.9961e+00,  1.0918e+00, -1.6562e+00,
         -3.2480e+00,  3.1714e-01,  8.8623e-01,  2.5757e-01, -4.5020e-01,
          3.1934e+00,  1.4600e+00, -1.3115e+00,  6.3916e-01, -9.0869e-01,
          7.8955e-01,  1.4453e+00, -5.3809e-01,  9.2529e-01,  1.9531e+00,
         -2.0195e+00,  3.8794e-01,  5.1904e-01, -5.9229e-01,  1.5308e-01,
          9.6436e-01,  1.0273e+00,  2.5293e+00, -2.6855e+00,  2.0391e+00,
         -1.3506e+00,  1.2373e+00, -2.3008e+00,  1.3457e+00,  4.5459e-01,
          2.3145e+00,  3.5400e-01,  3.7207e+00,  8.8672e-01, -4.6289e-01,
          1.5686e-01, -1.3965e+00,  1.8867e+00,  1.9238e+00, -2.2500e+00,
         -1.2291e-02,  4.0479e-01, -1.1680e+00,  3.0547e+00, -2.6270e+00,
          1.5107e+00,  1.8838e+00,  1.7412e+00, -8.7500e-01, -2.4727e+00,
          6.3379e-01,  2.7363e+00, -1.3115e+00,  7.8857e-01,  5.3320e-01,
         -7.6416e-01,  7.0508e-01,  1.9189e-01, -1.9373e-01,  1.6328e+00,
          5.5615e-01, -1.4121e+00,  2.2363e+00,  1.3623e+00, -5.9662e-02,
          1.0034e-01,  2.6440e-01, -1.8740e+00,  9.5215e-01,  3.2910e-01,
         -8.8037e-01,  1.0020e+00, -1.8604e+00,  1.8252e+00,  1.2817e-02,
         -3.9429e-01, -2.5898e+00,  2.0703e+00,  1.3342e-01,  2.0020e+00,
          1.5859e+00,  1.2520e+00,  1.0986e+00,  1.5996e+00,  7.7783e-01,
          9.6240e-01,  7.9883e-01,  1.3242e+00,  1.8835e-01,  2.6123e-01,
         -7.9443e-01, -4.3164e-01,  1.6113e+00,  3.1426e+00,  1.9736e+00,
         -2.1367e+00, -2.4011e-01, -2.8052e-01, -1.4727e+00, -1.7793e+00,
         -1.2500e+00,  1.5459e+00,  1.0215e+00, -5.3955e-01,  1.0576e+00,
          6.2451e-01, -3.3508e-02, -8.6572e-01, -7.8906e-01, -9.0088e-01,
         -1.9346e+00,  1.2285e+00, -9.6631e-01, -6.5674e-01,  9.4385e-01,
          1.4619e+00,  1.4697e+00, -1.5010e+00, -3.6426e-01,  1.1682e-01,
         -2.5410e+00,  1.1787e+00,  2.1460e-01, -3.0469e-01,  5.2490e-01,
          2.2949e+00, -8.1543e-02, -2.5195e+00,  2.7949e+00,  3.9624e-01,
         -1.8584e+00,  3.1372e-01, -1.3311e+00, -1.4209e-01,  5.6445e-01,
         -4.6997e-02,  7.8467e-01, -1.0947e+00,  2.9121e+00, -1.5557e+00,
         -1.4795e+00, -1.6660e+00,  9.5581e-02, -2.1472e-01, -2.4768e-01,
          1.7441e+00,  5.1318e-01,  7.5732e-01, -7.2510e-01,  6.2842e-01,
          3.3911e-01,  2.5952e-01,  6.9580e-01,  9.7314e-01,  3.4688e+00,
          4.6602e+00, -7.4829e-02, -2.5020e+00,  2.5342e-01, -9.0149e-02,
         -1.6870e-01,  1.4180e+00,  4.0039e+00,  5.8398e-01, -3.2013e-02,
          1.4434e+00, -3.4648e+00, -7.9736e-01,  1.2529e+00, -1.6373e-02,
         -2.3477e+00, -3.6816e-01, -4.6411e-01, -2.0007e-01, -1.2168e+00,
         -2.9062e+00, -9.2773e-01, -6.3330e-01,  5.8105e-01, -5.2686e-01,
          3.3789e-01, -6.3184e-01,  1.7383e+00,  1.0391e+00, -2.8945e+00,
         -3.4521e-01, -3.6353e-01, -8.4326e-01, -1.8838e+00,  3.0176e+00,
          2.3840e-01, -4.6295e-02,  7.5391e-01, -3.9429e-01,  1.1113e+00,
          1.8330e+00, -5.6787e-01, -1.2842e+00,  1.1572e+00, -5.6592e-01,
          4.4360e-01, -1.2695e+00,  2.5996e+00,  1.9355e+00, -5.8057e-01,
         -1.0654e+00, -4.7089e-02, -1.2627e+00,  2.2285e+00,  7.0333e-05,
          7.0654e-01, -1.5234e+00,  1.4580e+00, -3.2422e-01,  1.6494e+00,
         -5.3076e-01, -1.3037e+00,  1.0225e+00,  1.5107e+00, -2.7734e-01,
          3.2300e-01,  6.1328e-01, -3.1079e-01,  2.3711e+00, -2.0664e+00,
         -5.3662e-01, -7.9346e-01, -4.1064e-01, -5.9814e-01,  9.4727e-01,
          1.7461e+00,  5.3986e-02, -8.2275e-01, -1.8506e+00, -1.1699e+00,
         -1.2539e+00, -3.4180e+00, -2.3425e-01, -9.3506e-01,  1.1924e+00,
         -8.5547e-01, -5.6494e-01,  1.1560e-01, -1.5371e+00, -1.4717e+00,
         -8.4912e-01, -1.4395e+00,  1.3447e+00, -9.7217e-01, -1.3467e+00,
         -5.3320e-01, -2.7070e+00, -1.5215e+00, -1.3994e+00, -1.9795e+00,
         -5.9570e-01, -1.6279e+00, -4.8022e-01,  3.3667e-01, -3.1605e-03,
          1.0967e+00,  2.1641e+00,  8.6426e-01, -1.5605e+00, -1.6797e-01,
          1.2197e+00, -2.7090e+00, -1.3220e-01,  7.1875e-01, -1.1953e+00,
         -1.4980e+00,  2.7466e-01, -1.4648e+00,  4.5288e-02, -2.1797e+00,
         -3.4160e+00,  1.5430e-01,  1.0908e+00,  6.4209e-01, -3.5547e-01,
         -8.5547e-01,  4.0469e+00,  6.1462e-02,  3.1113e+00,  1.2451e+00,
          1.6895e+00,  1.1113e+00,  2.9668e+00,  2.0332e+00,  2.7246e-01,
          2.2109e+00,  7.5000e-01, -8.3923e-02,  3.5371e+00,  1.8193e+00,
          1.2119e+00,  3.3496e-01,  1.8877e+00,  1.4268e+00, -6.2744e-01,
          6.2061e-01, -5.1758e-01, -6.0645e-01, -3.8340e+00, -2.4160e+00,
         -1.7158e+00, -2.1582e+00,  3.5376e-01,  4.8682e-01,  1.0225e+00]],
       device='cuda:0', dtype=torch.float16, grad_fn=<AddmmBackward0>)

C:\Users\{UserName}>

@HolyWu Just wanted to ask you, did it work for you, without deleting "C:\Users{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\nvidia\cudnn" folder?

Yes. Although it throws multiple "entry point not found" errors about cuDNN DLLs, torch_compile_resnet_example still runs fine after clicking them away. You can of course pip uninstall -y nvidia-cudnn-cu12 to eliminate the errors.

The reason for the error is that, the version of cuDNN DLLs bundled in torch windows wheel is still 8.8.0.1. When you install tensorrt==9.2.0.post12.dev5, it installs the latest nvidia-cudnn-cu12 for you, which currently is 8.9.7.29. When you import torch, the 8.8.0.1 cuDNN DLLs got loaded. And now you import tensorrt, it tries to load cuDNN DLLs from nvidia-cudnn-cu12, then it explodes.

Thank you @HolyWu for the catch - that was a bug in my initial script which has now been addressed; apologies for that @jensdraht1999. Regarding the cuDNN issue, I'm looking for a workaround to make the imports smoother - potentially by having one of torch or tensorrt just utilize the other's cuDNN version.

@jensdraht1999 - it is good to see the torch.compile example is working and Torch-TensorRT should work on your GPU. I am wondering if you are seeing any issues now when running the torch_compile_resnet_example? Does the ERROR RUNNING GUARDS forward and AttributeError: module 'torch._dynamo.eval_frame' has no attribute 'cached_backends' still persist?

@gs-olive Reinstalling everything again, because it seems, I had the TensorRT Folder in the path, which might have caused errors. Will tell you everything when done.

Run as non-admin and also as Admin per CMD.

GPU Hardware sceduling on/off tested. Currently off.

Energy Profile Performance and Powersave both tested.

Windows restarted also after installation.

The file:"eval_frame.py" from the link posted by you, has been put in the folder:"C:\Users{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\torch_dynamo\" Replaced the file. Also check, if this file is different. Yes, it is. This is the file (here uploaded txt file, but of course in the folder it is a .py file): eval_frame.txt

I can confirm this will exists and gets downloaded again, if non-eexistend: C:\Users{UserName}.cache\torch\hub\checkpoints\resnet18-f37072fd.pth

Nvidia Cuda System Fallback Policy: Default, but also tested ON/OFF, the same.

Everything from the path is in the jpg here below: Enviromentvariable

This is the system enviroment: Envirment2

This happens, if I run "import torch; import torch_tensorrt":

Windows PowerShell
Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten.

Lernen Sie das neue plattformübergreifende PowerShell kennen – https://aka.ms/pscore6

PS C:\Users\{UserName}> python
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch; import torch_tensorrt
C:\Users\{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\_Device.py:19: UserWarning: Unable to import torchscript frontend core and torch-tensorrt runtime. Some dependent features may be unavailable.
  warnings.warn(
>>>

And then your example: torch_compile_resnet_example.py with this content:

Please look this file:

torch_compile_resnet_example.txt

WITH THE FOLDER "C:\Users{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\nvidia\cudnn" following happens: DLL ERRORs which you can click away, then it will work. Cuda is working according to Taskmanager. log1.txt please look at it. log1.txt

WITHOUT THE FOLDER: C:\Users{UserName}\AppData\Local\Programs\Python\Python310\Lib\site-packages\nvidia\cudnn No DLL Errors and also the same log. Cuda is working according to Taskmanager. log2.txt please look at it. log2.txt

Also this here happens with this file with following content (Holywu corrected example):

import torch import torchvision.models as models

model = models.resnet18(pretrained=True).half().eval().to("cuda") inputs = [torch.randn((1, 3, 224, 224)).to("cuda").half()] optimized_model = torch.compile(model, backend="eager") print(optimized_model(*inputs))

log3.txt please look at it. log3.txt

@gs-olive I wonder where the compiled model is stored or the workspace is? And also what exactly should happen in torch_compile_resnet_example.py?

Hi @jensdraht1999 - thanks for testing this out - these logs are very helpful.

Normally, the compiled model is stored in the torch.compile context, which means Torch caches the tensorrt engines once they have been generated by Torch-TensorRT. The torch_compile_resnet_example.py should just compile the model (twice I believe), then exit without any errors. In your sample, it almost works, but fails in eval_frame.py

My guess is that the issue with the torch_compile_resnet_example.py is that the eval_frame.py version being used is stale. That PR which I referenced has not been rebased in awhile, and the eval_frame.py code may no longer be valid for the latest main. I made a new PR which has that same change applied to the latest version of pytorch main, here: link + go to eval_frame.py. Alternatively, you can directly apply the code change to whatever version of eval_frame.py came with the distribution of Torch you are using.

@gs-olive

I need to install this here, which you had installed.: torch 2.3.0.dev20240116+cu121 torchvision 0.18.0.dev20240116+cu121

This should be the exact version you already have: https://download.pytorch.org/whl/nightly/cu121/torchvision-0.18.0.dev20240116%2Bcu121-cp310-cp310-win_amd64.whl https://download.pytorch.org/whl/nightly/cu121/torch-2.3.0.dev20240116%2Bcu121-cp310-cp310-win_amd64.whl

My current installation is like this: black 23.12.1 certifi 2023.11.17 cfgv 3.4.0 charset-normalizer 3.3.2 clang-format 14.0.6 click 8.1.7 colorama 0.4.6 distlib 0.3.8 exceptiongroup 1.2.0 expecttest 0.2.1 filelock 3.13.1 fsspec 2023.12.2 huggingface-hub 0.20.3 identify 2.5.33 idna 3.6 iniconfig 2.0.0 intel-openmp 2021.4.0 isort 5.13.2 Jinja2 3.1.3 MarkupSafe 2.1.4 mkl 2021.4.0 mpmath 1.3.0 mypy 1.8.0 mypy-extensions 1.0.0 networkx 3.2.1 nodeenv 1.8.0 numpy 1.26.3 nvidia-cublas-cu12 12.3.4.1 nvidia-cuda-nvrtc-cu12 12.3.107 nvidia-cuda-runtime-cu12 12.3.101 nvidia-cudnn-cu12 8.9.7.29 packaging 23.2 parameterized 0.9.0 pathspec 0.12.1 pillow 10.2.0 pip 23.3.2 platformdirs 4.1.0 pluggy 1.4.0 pre-commit 3.6.0 pytest 7.4.4 PyYAML 6.0.1 regex 2023.12.25 requests 2.31.0 ruff 0.1.14 safetensors 0.4.2 setuptools 65.5.0 sympy 1.12 tbb 2021.11.0 tensorrt 9.2.0.post12.dev5 tensorrt-bindings 9.2.0.post12.dev5 tensorrt-libs 9.2.0.post12.dev5 timm 0.9.12 tokenizers 0.15.1 tomli 2.0.1 torch 2.3.0.dev20240124+cu121 torch-tensorrt 2.3.0.dev0+e7e291144 torchvision 0.18.0.dev20240124+cpu tqdm 4.66.1 transformers 4.37.1 typing_extensions 4.9.0 urllib3 2.1.0 virtualenv 20.25.0 wheel 0.42.0

I also applied the patch you have provided, it was unforunately worse, because it made a error. I only upaded the eval_frame.py file.

The log here:

log.txt

@gs-olive I will use the exact version you have also used and try again. I will inform you about it.

And if it's working for you and Holywu and the other user in this thread, than this problem must be somehow my setup.

@gs-olive

I have installed an old version torch and torchvision. It still does not work.

My installed packagages: attrs 23.2.0 black 23.12.1 certifi 2023.11.17 cffi 1.16.0 cfgv 3.4.0 charset-normalizer 3.3.2 clang-format 14.0.6 click 8.1.7 colorama 0.4.6 distlib 0.3.8 et-xmlfile 1.1.0 exceptiongroup 1.2.0 expecttest 0.2.1 filelock 3.13.1 fsspec 2023.12.2 h11 0.14.0 huggingface-hub 0.20.3 identify 2.5.33 idna 3.6 iniconfig 2.0.0 intel-openmp 2021.4.0 isort 5.13.2 Jinja2 3.1.3 MarkupSafe 2.1.4 mkl 2021.4.0 mpmath 1.3.0 mypy 1.8.0 mypy-extensions 1.0.0 networkx 3.2.1 nodeenv 1.8.0 numpy 1.26.3 nvidia-cublas-cu12 12.3.4.1 nvidia-cuda-nvrtc-cu12 12.3.107 nvidia-cuda-runtime-cu12 12.3.101 nvidia-cudnn-cu12 8.9.7.29 openpyxl 3.1.2 outcome 1.3.0.post0 packaging 23.2 parameterized 0.9.0 pathspec 0.12.1 pillow 10.2.0 pip 23.3.2 platformdirs 4.1.0 pluggy 1.4.0 pre-commit 3.6.0 pycparser 2.21 PySocks 1.7.1 pytest 7.4.4 PyYAML 6.0.1 regex 2023.12.25 requests 2.31.0 ruff 0.1.14 safetensors 0.4.2 selenium 4.17.2 setuptools 65.5.0 sniffio 1.3.0 sortedcontainers 2.4.0 sympy 1.12 tbb 2021.11.0 tensorrt 9.2.0.post12.dev5 tensorrt-bindings 9.2.0.post12.dev5 tensorrt-libs 9.2.0.post12.dev5 timm 0.9.12 tokenizers 0.15.1 tomli 2.0.1 torch 2.3.0.dev20231214+cu121 torch-tensorrt 2.3.0.dev0+e7e291144 torchvision 0.18.0.dev20240116+cu121 tqdm 4.66.1 transformers 4.37.1 trio 0.24.0 trio-websocket 0.11.1 typing_extensions 4.9.0 urllib3 2.1.0 virtualenv 20.25.0 wheel 0.42.0 wsproto 1.2.0

1.log

So it seems it must be a error on my side somehow. I'll have to look.

7. python setup.py install

Thanks for your input, I have tested your steps and they do not work on my Windows machine. When I try to run setup.py it gives me an error, saying I need to specify the commands. Any ideas?

@enislalmi What exactly it does say?

work in some compilation cases since it is under active de

Hi, I got it downloading with your way, but I get this error RuntimeError: Tried to instantiate class 'tensorrt.Engine', but it does not exist! Ensure that it is registered via torch::class_

@enislalmi What exactly it does say? Hi, i got it running with the other version that was specified below your comment, but now I have this errorç RuntimeError: Tried to instantiate class 'tensorrt.Engine', but it does not exist! Ensure that it is registered via torch::class_, any ideas? Thanks!

Before everything install Cuda 12 version.

Do you have git installed, because after installing it, you need to do this: Clone the branch: git clone --single-branch --branch windows_CI https://github.com/pytorch/TensorRT.git

And then the rest of this: cd into the directory containing the cloned repo. For instance, cd TensorRT

Run python setup.py install to install torch_tensorrt

Open a Python interpreter and run import torch; import torch_tensorrt.

There may be warnings about TorchScript and Torchvision, but the import should succeed.

@jensdraht1999 yes everything is done as you said, but I still receive the same error. Thanks!

Hi @gs-olive. I modified torch_compile_resnet_example.py to use ir="dynamo" instead of ir="torch_compile". But it gave the following error. ir="torch_compile" works fine here.

[02/16/2024-00:39:27] [TRT] [E] 3: [executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2082] Error Code 3: API Usage Error (Parameter check failed at: executionContext.cpp::nvinfer1::rt::ExecutionContext::validateInputBindings::2082, condition: profileMaxDims.d[i] >= dimensions.d[i]. Supplied binding dimension [8,3,224,224] for bindings[0] exceed min ~ max range at index 0, maximum dimension in profile is 1, minimum dimension in profile is 1, but supplied dimension is 8.

Full log: ir_dynamo.log

By the way, I couldn't find a page on https://pytorch.org/TensorRT explaining the key difference between ir="dynamo" and ir="torch_compile", what's the pros/cons of each, and which one is recommended or gives the best performance.

@enislalmi If import did work as it should and the example provided by Gsolive did work, then it did work.

You have to be aware, that currently this is in development and not everything is guaranteed to work.

Hi @HolyWu - thanks for testing that out! This is actually expected in this case, since ir="dynamo" does not recompile upon detecting new input shapes. Dynamic shapes need to be specified via the Input(...) API, as here. I just tested it out though and it seems that Dynamic shapes in ir="dynamo" on Windows is not yet functional.

Below is a brief summary of the different ir choices:

`ir="torch_compile"`

Uses torch.compile + handles complex Python logic/code using Torch Dynamo
Primarily for Just-In-Time usecases and useful for quick prototyping + testing of Torch-TensorRT
Does not return a portable artifact (yet)
Only Python deployment
Currently supports Dynamic shapes by recompiling the model (to be addressed soon)

`ir="dynamo"`

Works with torch.export + ideal for cases where model serialization is needed
Primarily for Ahead-Of-Time usecases, serializable, supports Dynamic shapes without recompilation
Supports Python or C++ deployment

Both paths share a lot of implementation logic and have similar performance. For instance, the converter implementations translating Torch to TensorRT are common between the paths.

@gs-olive I tried again, somehow there could be any problem. What exactly do you get for the output for the example you had provided. Here is my output:

[03/03/2024-23:16:52] [TRT] [V] CUDA lazy loading is enabled. DEBUG:torch_tensorrt.dynamo._DryRunTracker: ++++++++++++++++++++++++++++++++++++++++++++++++++ Dry-Run Results for Graph ++++++++++++++++++++++++++++++++++++++++++++++++++

The graph consists of 89 Total Operators, of which 89 operators are supported, 100.0% coverage

Compiled with: CompilationSettings(precision=torch.float16, debug=True, workspace_size=21474836480, min_block_size=7, torch_executed_ops={}, pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=True, truncate_long_and_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=False, disable_tf32=False, sparse_weights=False, refit=False, engine_capability=<EngineCapability.DEFAULT: 0>, num_avg_timing_iters=1, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912, dryrun=False, hardware_compatible=False)

Graph Structure:

Inputs: List[Tensor: (1, 3, 224, 224)@float16] ... TRT Engine #1 - Submodule name: _run_on_acc_0 Engine Inputs: List[Tensor: (1, 3, 224, 224)@float16] Number of Operators in Engine: 89 Engine Outputs: Tensor: (1, 1000)@float16 ... Outputs: Tuple(Tensor: (1, 1000)@float16)

------------------------- Aggregate Stats -------------------------

Average Number of Operators per TRT Engine: 89.0 Most Operators in a TRT Engine: 89

** Recommendations **

For minimal graph segmentation, select min_block_size=89 which would generate 1 TRT engine(s)
The current level of graph segmentation is equivalent to selecting min_block_size=89 which generates 1 TRT engine(s) [03/03/2024-23:16:52] [TRT] [W] Using default stream in enqueue()/enqueueV2()/enqueueV3() may lead to performance issues due to additional cudaDeviceSynchronize() calls by TensorRT to ensure correct synchronizations. Please use non-default stream instead. ERROR RUNNING GUARDS forward C:\Users{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torchvision\models\resnet.py:284 lambda L, G, _kwargs_ignored: _guardedcode.valid and check_global_state() and hasattr(L['x'], '_dynamo_dynamic_indices') == False and ___check_obj_id(L['self'], 616323280) and L['self'].training == False and utils_device.CURRENTDEVICE == None and (skip_backend_check() or _current_backend() == _lookupbackend(712378160)) and check_type_id(G['import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks, 140722909523296) and list(G['import_torch_dot_nn_dot_modules_dot_module']._global_forwardhooks.keys()) == [] and check_type_id(G['import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks, 140722909523296) and list(G['import_torch_dot_nn_dot_modules_dot_module']._global_backward_hooks.keys()) == [] and _check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks, 140722909523296) and list(G['import_torch_dot_nn_dot_modules_dot_module']._global_forward_pre_hooks.keys()) == [] and _check_type_id(G['__import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks, 140722909523296) and list(G['import_torch_dot_nn_dot_modules_dot_module']._global_backward_pre_hooks.keys()) == [] and ___check_tensors(L['x'], tensor_check_names=tensor_check_names) Traceback (most recent call last): File "C:\Users{UserName}\Desktop\whl\torch_compile_resnet_example.py", line 76, in new_batch_size_outputs = optimized_model(new_batch_size_inputs) File "C:\Users{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "C:\Users{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl return forward_call(*args, kwargs) File "C:\Users{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_dynamo\eval_frame.py", line 410, in _fn return fn(*args, *kwargs) File "C:\Users{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "C:\Users{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) File "", line 17, in guard File "C:\Users{UserName}\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_dynamo\guards.py", line 104, in lambda backend_obj_id: torch._dynamo.eval_frame.cached_backends.get( AttributeError: module 'torch._dynamo.eval_frame' has no attribute 'cached_backends' Unexpected Internal Error: [virtualMemoryBuffer.cpp::nvinfer1::StdVirtualMemoryBufferImpl::~StdVirtualMemoryBufferImpl::109] Error Code 1: Cuda Driver (TensorRT internal error)

Thanks for the follow-up. For the example I had referenced, it runs to completion on my machine with no errors. Not sure if this will help, but there was a recent Windows-support PR I added to Torch recently which was merged https://github.com/pytorch/pytorch/pull/115969. Upgrading to the latest nightly version of Torch could be helpful with this, since the AttributeError: module 'torch._dynamo.eval_frame' has no attribute 'cached_backends' as shown in your logs should no longer be occurring on the latest main, since that attribute is present here.

pip3 install --pre -U torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu121

Ok I will do it, but i wonder if this can work, because the PR was closed, not merged. Are you sure if I run this, it will work?

pip3 install --pre -U torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu121

I will have to wait to go home to install this. But thank you very much for the help.

I think the logs from this run will still be helpful for debugging if a different error appears. Even though it shows as "Closed", the changes are reflected on main, so it seems to have been merged.

pytorch / TensorRT