search

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM
Apache License 2.0
8.47k stars 957 forks source link

trtllm-bench "No module named 'tensorrt_llm.bench.datamodels'" in v0.13.0 #2347

Open activezhao opened 5 days ago

activezhao commented 5 days ago

System Info

CPU x86_64

GPU NVIDIA L20

TensorRT branch: v0.13.0

CUDA: NVIDIA-SMI 535.161.07 Driver Version: 535.161.07 CUDA Version: 12.5

Who can help?

@kaiyux @byshiue

Information

Tasks

Reproduction

I want to use trtllm-bench for perf tests, and I launch nvcr.io/nvidia/tritonserver:24.09-trtllm-python-py3.

But I got error , when I execute the command like this:

trtllm-bench --model /data/deepseek-6.7b build --tp_size 2 --quantization FP8 --dataset /data/perf_dataset/requests_num_1000_input_2048_output_32.txt

The error is:

[TensorRT-LLM] TensorRT-LLM version: 0.13.0
Traceback (most recent call last):
  File "/usr/local/bin/trtllm-bench", line 5, in <module>
    from tensorrt_llm.commands.bench import main
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/bench.py", line 7, in <module>
    from tensorrt_llm.bench.run.run import run_command
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/bench/run/run.py", line 17, in <module>
    from tensorrt_llm.bench.datamodels import BenchmarkStatistics
ModuleNotFoundError: No module named 'tensorrt_llm.bench.datamodels'

Expected behavior

The trtllm-bench command works.

actual behavior

the tensorrt-llm info is this:

tensorrt           10.4.0
tensorrt-llm       0.13.0

The command is this:

trtllm-bench --model /data/deepseek-6.7b build --tp_size 2 --quantization FP8 --dataset /data/perf_dataset/requests_num_1000_input_2048_output_32.txt

The error is:

[TensorRT-LLM] TensorRT-LLM version: 0.13.0
Traceback (most recent call last):
  File "/usr/local/bin/trtllm-bench", line 5, in <module>
    from tensorrt_llm.commands.bench import main
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/bench.py", line 7, in <module>
    from tensorrt_llm.bench.run.run import run_command
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/bench/run/run.py", line 17, in <module>
    from tensorrt_llm.bench.datamodels import BenchmarkStatistics
ModuleNotFoundError: No module named 'tensorrt_llm.bench.datamodels'

additional notes

Please help me analyze this problem.

Thanks.

valent33 commented 5 days ago

Replacing this line worked for me: from tensorrt_llm.bench.datamodels import BenchmarkStatistics to from tensorrt_llm.bench.run.dataclasses import BenchmarkStatistics

activezhao commented 4 days ago

Replacing this line worked for me: from tensorrt_llm.bench.datamodels import BenchmarkStatistics to from tensorrt_llm.bench.run.dataclasses import BenchmarkStatistics

@valent33 It works!

Thank you for your suggestion.