Open ttim opened 9 months ago
Thanks for the request, @ttim! We are working on a more general quantization tool to work with any TRT-LLM model so we don't have to worry about individual model support. It will take us some time do so, but keep an eye out next year!
Currently building StarCoder GPT variant fails if smooth quant applied.
Is it planned to be supported? Any advice on how to build it?