search

AniZpZ / AutoSmoothQuant

An easy-to-use package for implementing SmoothQuant for LLMs
MIT License
82 stars 7 forks source link

是否支持int4量化 #23

Open AlexMa0 opened 4 months ago

AlexMa0 commented 4 months ago

autosmoothquant是只支持int8的量化吗？是否可以支持int4的量化？

AniZpZ commented 4 months ago

You can use SmoothQuant to implement w4a8 quantization, but this may result in a non-negligible loss of model performance. If you are interested in performing w4a8 quantization for inference, you can refer to our new project QQQ.