Feature request: Support SmoothQuant variant of StarCoder

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

https://nvidia.github.io/TensorRT-LLM

Apache License 2.0

7.97k stars 869 forks source link

Feature request: Support SmoothQuant variant of StarCoder #324

Open ttim opened 9 months ago

ttim commented 9 months ago

Currently building StarCoder GPT variant fails if smooth quant applied.

Is it planned to be supported? Any advice on how to build it?

ncomly-nvidia commented 8 months ago

Thanks for the request, @ttim! We are working on a more general quantization tool to work with any TRT-LLM model so we don't have to worry about individual model support. It will take us some time do so, but keep an eye out next year!