Friction Report: Using TensorRT-LLM on Windows

[dan-homebrew]

Overview

• This is my friction log for getting started with TensorRT-LLM on Windows
• From POV of non-ML engineer vaguely familiar with architecture

Current Status

• [x] Installed all dependencies from based on Nvidia Windows Setup
  • [x] Ran commands manually as Powershell script failed without message
  • [x] Validated TensorRT install by running python -c "import tensorrt as trt; print(trt.__version__)"
  • [ ] Unable to validate TensorRT-LLM install by running python -c "import tensorrt_llm; print(tensorrt_llm._utils.trt_version())"

PS C:\Users\dan\Github\tensorrt-llm\windows> python -c "import tensorrt_llm; print(tensorrt_llm._utils.trt_version())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python310\lib\tensorrt_llm\__init__.py", line 30, in <module>
    import tensorrt_llm.functional as functional
  File "C:\Program Files\Python310\lib\tensorrt_llm\functional.py", line 29, in <module>
    from . import graph_rewriting as gw
  File "C:\Program Files\Python310\lib\tensorrt_llm\graph_rewriting.py", line 12, in <module>
    from .network import Network
  File "C:\Program Files\Python310\lib\tensorrt_llm\network.py", line 26, in <module>
    from tensorrt_llm.module import Module
  File "C:\Program Files\Python310\lib\tensorrt_llm\module.py", line 17, in <module>
    from ._common import default_net
  File "C:\Program Files\Python310\lib\tensorrt_llm\_common.py", line 32, in <module>
    from .plugin import _load_plugin_lib
ModuleNotFoundError: No module named 'tensorrt_llm.plugin'

# Note: pip shows TensorRT-LLM as being installed
PS C:\Users\dan\Github\tensorrt-llm\windows> pip show tensorrt_llm
Name: tensorrt-llm
Version: 0.8.0
Summary: TensorRT-LLM: A TensorRT Toolbox for Large Language Models
Home-page: https://github.com/NVIDIA/TensorRT-LLM
Author: NVIDIA Corporation
Author-email:
License: Apache License 2.0
Location: c:\program files\python310\lib\site-packages
Requires: accelerate, build, colored, cuda-python, diffusers, evaluate, janus, mpi4py, numpy, onnx, optimum, polygraphy, psutil, pynvml, pywin32, sentencepiece, tensorrt, tokenizers, torch, torchdata, torchtext, torchvision, transformers, wheel
Required-by:
PS C:\Users\dan\Github\tensorrt-llm\windows>

Debugging Log

• [x] Search pip install issues with TensorRT-LLM
  • [x] Tried installing wheel from #1152

Path

• [ ] Install TensorRT-LLM
• [ ] Convert Mistral 7b to TensorRT-LLM following Llama Example

[tikikun]

stale? feel free to reopen if needed