search

NVIDIA / TensorRT-Model-Optimizer

TensorRT Model Optimizer is a unified library of state-of-the-art model optimization techniques such as quantization, pruning, distillation, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM or TensorRT to optimize inference speed on NVIDIA GPUs.
https://nvidia.github.io/TensorRT-Model-Optimizer
Other
574 stars 43 forks source link

Whether the comfyui_tensorrt node can load the tensorrt plan model generated by the tool ? #85

Open blacklong28 opened 1 month ago

blacklong28 commented 1 month ago

I used the tool to convert an SDXL model to generate the tensorrt model (.plan), and then I tried to load it using Comfyui and plug it into the sampler's MODEL input, but the following error was reported：

10/11/2024-13:32:07] [TRT] [I] Loaded engine size: 5324 MiB
[10/11/2024-13:32:07] [TRT] [I] [MS] Running engine with multi stream info
[10/11/2024-13:32:07] [TRT] [I] [MS] Number of aux streams is 2
[10/11/2024-13:32:07] [TRT] [I] [MS] Number of total worker streams is 3
[10/11/2024-13:32:07] [TRT] [I] [MS] The main stream provided by execute/enqueue calls is the first worker stream
[10/11/2024-13:32:10] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +3112, now: CPU 0, GPU 8365 (MiB)
model_type EPS
#10 [TensorRTLoader]: 7.20s
Requested to load SDXL
Loading 1 new model
loaded completely 0.0 0.0 True
  0%|                                                                                                                                                       | 0/20 [00:00<?, ?it/s]
[10/11/2024-13:32:10] [TRT] [E] IExecutionContext::setInputShape: Error Code 3: Internal Error (Given invalid tensor name: x. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] IExecutionContext::setInputShape: Error Code 3: Internal Error (Given invalid tensor name: timesteps. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] IExecutionContext::setInputShape: Error Code 3: Internal Error (Given invalid tensor name: context. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] IExecutionContext::setInputShape: Error Code 3: Internal Error (Given invalid tensor name: y. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] ICudaEngine::getTensorDataType: Error Code 3: Internal Error (Given invalid tensor name: x. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] ICudaEngine::getTensorDataType: Error Code 3: Internal Error (Given invalid tensor name: timesteps. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] ICudaEngine::getTensorDataType: Error Code 3: Internal Error (Given invalid tensor name: context. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] ICudaEngine::getTensorDataType: Error Code 3: Internal Error (Given invalid tensor name: y. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] IExecutionContext::setTensorAddress: Error Code 3: Internal Error (Given invalid tensor name: x. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] IExecutionContext::setTensorAddress: Error Code 3: Internal Error (Given invalid tensor name: timesteps. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] IExecutionContext::setTensorAddress: Error Code 3: Internal Error (Given invalid tensor name: context. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] IExecutionContext::setTensorAddress: Error Code 3: Internal Error (Given invalid tensor name: y. Get valid tensor names with getIOTensorName())
[10/11/2024-13:32:10] [TRT] [E] IExecutionContext::enqueueV3: Error Code 3: API Usage Error (Parameter check failed, condition: mContext.profileObliviousBindings.at(profileObliviousIndex) != nullptr. Address is not set for input tensor sample. Call setInputTensorAddress or setTensorAddress before enqueue/execute.)
  0%|                                                                                                                                                       | 0/20 [00:00<?, ?it/s]
!!! Exception during processing !!! The size of tensor a (128) must match the size of tensor b (6) at non-singleton dimension 3
...

The cmd convert from onnx to trt I used:

/usr/src/tensorrt/bin/trtexec --builderOptimizationLevel=4 --stronglyTyped --onnx=./backbone.onnx   --minShapes=sample:2x4x128x128,timestep:1,encoder_hidden_states:2x77x2048,text_embeds:2x1280,time_ids:2x6   --optShapes=sample:16x4x128x128,timestep:1,encoder_hidden_states:16x77x2048,text_embeds:16x1280,time_ids:16x6   --maxShapes=sample:16x4x128x128,timestep:1,encoder_hidden_states:16x77x2048,text_embeds:16x1280,time_ids:16x6   --saveEngine=backbone.plan

Can I use the backbone.plan model Load by Comfyui (https://github.com/comfyanonymous/ComfyUI_TensorRT) TensorRT Loader Node ? Can someone help me? Thank you！