TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
Traceback (most recent call last):
File "/data/projects/tensorRT-LLM-test/build_engine/llama/build.py", line 983, in <module>
build(0, args)
File "/data/projects/tensorRT-LLM-test/build_engine/llama/build.py", line 927, in build
engine = build_rank_engine(builder, builder_config, engine_name,
File "/data/projects/tensorRT-LLM-test/build_engine/llama/build.py", line 777, in build_rank_engine
load_from_binary(tensorrt_llm_llama,
File "/data/miniconda3/envs/TensorRT-LLM-cu122-cpp/lib/python3.10/site-packages/tensorrt_llm/models/llama/weight.py", line 1019, in load_from_binary
t = fromfile(
File "/data/miniconda3/envs/TensorRT-LLM-cu122-cpp/lib/python3.10/site-packages/tensorrt_llm/models/llama/weight.py", line 901, in fromfile
t = t.reshape(shape)
ValueError: cannot reshape array of size 66060288 into shape (7168,8960)
additional notes
I found that this error occured in loading file ./engines/yi/sq05/1-gpu/model.layers.0.attention.query_key_value.weight.int8.col.0.bin
In addition, do smoothquant in TP=4 or TP=1 would both got this error
System Info
-GPU: 4 * 3090(24G) -TensorRT-LLM version: 0.7.1, built from source released last week -TensorRT version: 9.2.0.post12.dev5 -Nvidia Driver: Driver Version: 535.54.03 CUDA Version: 12.2 -OS: Ubuntu 20.04
Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Expected behavior
The engine should be built successfully
actual behavior
Got error msg:
additional notes
I found that this error occured in loading file ./engines/yi/sq05/1-gpu/model.layers.0.attention.query_key_value.weight.int8.col.0.bin
In addition, do smoothquant in TP=4 or TP=1 would both got this error