How can I use the trex tool to analyze the engine generated by trtllm

[BugCrown]

Description

I generated Phi-3-mini-4K-instruct engine based on ./example/phi/README.md and wanted to analyze it by trt-enging-explorer in trt. However, trex told me getPluginCreator could not find plugin: Gemmtensorrt_llm version: 1. I tried to add all of the .so files in ./tensorrt_llm/libs to trtexec's plugin library but it still didn't work. The command line is as follows：

plugins=$(echo /app/tensorrt_llm/lib/*.so | sed 's/ / --plugins=/g')
plugins="--plugins=$plugins"
trtexec --loadEngine=./phi-engine/rank0.engine $plugins

Equivalent to：

trtexec --loadEngine=./phi-engine/rank0.engine --plugins=/app/tensorrt_llm/lib/libnvinfer_plugin_tensorrt_llm.so --plugins=/app/tensorrt_llm/lib/libtensorrt_llm.so --plugins=/app/tensorrt_llm/lib/libtensorrt_llm_nvrtc_wrapper.so --plugins=/app/tensorrt_llm/lib/libth_common.so

The error is as follows：

[E] [TRT] 3: getPluginCreator could not find plugin: GPTAttentiontensorrt_llm version: 1
Error[1]: [pluginV2Runner.cpp::load::294] Error Code 1: Serialization (Serialization assertion creator failed.Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)

How can I add plugins from trtllm to trex?

Environment

TensorRT Version:10.0.1

NVIDIA GPU:RTX 4090

NVIDIA Driver Version: 535.183.01

CUDA Version:12.2

CUDNN Version:V12.4.99

Operating System:Ubuntu 22.04.4 LTS

Python Version (if applicable):3.10.12

Tensorflow Version (if applicable):

PyTorch Version (if applicable):2.3.0a0+40ec155e58.nv24.03

Baremetal or Container (if so, version):

[lix19937]

How do you generate rank0.engine ? What is your cmd ?

[BugCrown]

How do you generate rank0.engine ? What is your cmd ?

Here is the cmd:

export MODEL_TYPE="Phi-3-mini-4k-instruct"
python ./convert_checkpoint.py --model_type ${MODEL_TYPE} \
                    --model_dir "microsoft/${MODEL_TYPE}" \
                    --output_dir ./phi-checkpoint \
                    --dtype float16

trtllm-build \
    --checkpoint_dir ./phi-checkpoint \
    --output_dir ./phi-engine \
    --max_batch_size 8 \
    --max_input_len 1024 \
    --max_output_len 1024 \
    --tp_size 1 \
    --pp_size 1

You can find it in TensorRT-LLM/examples/phi/README.md

[lix19937]

[E] [TRT] 3: getPluginCreator could not find plugin: GPTAttentiontensorrt_llm version: 1

plugin name GPTAttentiontensorrt_llm not match. ref https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/plugins/gptAttentionPlugin/gptAttentionPlugin.cpp#L41 plugin name is GPTAttention

[BugCrown]

I know where gptattention is in trtllm. But how can I add this plugin to the trex library? Obviously .cpp files cannot be added as libraries. And I would like to know . /tensorrt_llm/libs if the compiled plugin library for trtllm already exists.

[lix19937]

But how can I add this plugin to the trex library

LIke your method, --plugins=

[BugCrown]

I tried. At the beginning of this issue, I added the .so files to the plugin library. But it didn't work. I want to know whether these .so files really contain gptAttention and other operator plugins. If not, how to compile these plugins into libraries.

[lix19937]

nm -D -A  -C /app/tensorrt_llm/lib/libnvinfer_plugin_tensorrt_llm.so  | grep " T "   |grep GPTAttentionPlugin   

will see gptAttentionPlugin is into compile.

Then you can define a onnx include GPTAttention node, and use

trtexec --onnx=${onnx} --verbose --plugins=/app/tensorrt_llm/lib/libnvinfer_plugin_tensorrt_llm.so 

[BugCrown]

Does it mean that if the object of trex is .engine, then the plugin cannot be inserted?