search

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM
Apache License 2.0
7.63k stars 832 forks source link

Import Error: ModuleNotFoundError: No module named 'tensorrt_llm.lora_manager' #1289

Open hademircii opened 4 months ago

hademircii commented 4 months ago

System Info

started receiving this error after rebuilding image today - that I believe picked up #1274 . (using pip3 install tensorrt_llm -U --pre --extra-index-url https://pypi.nvidia.com). 

Traceback (most recent call last):
  File "/TensorRT-LLM/examples/enc_dec/run.py", line 32, in <module>
    from tensorrt_llm.lora_manager import LoraManager
ModuleNotFoundError: No module named 'tensorrt_llm.lora_manager'

Who can help?

kaiyux

Information

Tasks

Reproduction

follow the readme here https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/enc_dec/README.md#run to run line: https://github.com/NVIDIA/TensorRT-LLM/blob/4bb65f216f2f1dbddb022f4fdd8925c2856baa58/examples/enc_dec/run.py#L32

Expected behavior

import sucessfull.

actual behavior

import fails.

additional notes

-

jagadeesh-k-ti commented 4 months ago

same issue. is there any known workaround?

hademircii commented 4 months ago

a simple workaround is to work with a previous commit to main, right before #1274 ..

luvwinnie commented 4 months ago

same here. trying this link.

https://console.brev.dev/notebook/streamingllm-tensorrt-llm

byshiue commented 4 months ago

Could you share the tensorrt_llm version you use? I try to reproduce but fail. 

bhsueh@xxx:/tmp/TensorRT-LLM$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorrt_llm.lora_manager import LoraManager
[TensorRT-LLM] TensorRT-LLM version: 0.9.0.dev2024031200
luvwinnie commented 4 months ago

I'm using this version.

>>> import tensorrt_llm
No protocol specified
[TensorRT-LLM] TensorRT-LLM version: 0.8.0

I think I should upgrade to latest and take a try.

RankoR commented 2 months ago

I had this issue when was using 0.9.0 version installed from pip over 0.8.0 version of this repo. Updating repo to 0.9.0 fixed it.

github-actions[bot] commented 1 month ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."