search

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs
MIT License
3.22k stars 238 forks source link

How to improve the performance of gemm_half_q_half_gptq_kernel operator? #297

Open lovelynight opened 5 months ago

lovelynight commented 5 months ago

On the A100 graphics card, gptq's custom operator has been tested and is slower than pytorch's linear layer when batch > 10, thus affecting the throughput of high concurrency during model inference. I tried to modify the hyperparameters in q_gemm, but there was no obvious improvement. I would like to ask if there are any ideas to improve the speed of high concurrency? Thanks

turboderp commented 5 months ago

I am working on some improvements, yes, specifically for batch sizes around 5-30, but the improvements should also make it easier to adjust kernel parameters to optimize for other architectures. I'm not sure when I'll have anything ready, though.

(Basically the issue is that the kernel becomes compute-bound at higher batch sizes, and the solution is likely going to be tensor cores.)