search

bytedance / ABQ-LLM

An acceleration library that supports arbitrary bit-width combinatorial quantization operations
Apache License 2.0
226 stars 25 forks source link

CUDA kernel of weight only quantization #13

Open Sekri0 opened 1 month ago

Sekri0 commented 1 month ago

As mentioned in README, [Note that due to the limitations of AutoGPTQ kernels, the real quantization of weight-only quantization can only lead memory reduction, but with slower inference speed.] I'm a little confused. Does this mean that ABQ-LLM's weight-only quantization directly reuses GPTQ's cuda kernel？

zengchao0424 commented 1 month ago

Thanks for your attention to our work. Matrix multiplication of int and float is not supported, but based on our experience in model optimization, the effect of int16 and float16 will be basically aligned (sd or llm).

So I suggest you try W2Aint16. In this case, you can directly use our operator for acceleration. Our operator is suitable for W2 scenarios.