TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
I refer to this Issue and want to use my own data set to obtain the scale value of SQ.
The scenario is 70b tp=2, the length of input_ids is not limited, instead of the default 923 in the script
But currently OOM occurs when the model is forwarded. I guess it is using the HF interface that causes OOM.
Is there any solution for this? Thanks~
Specific orders: