Open juanps90 opened 1 year ago
Thanks! I'll look into it
tested it and found 50% performance up. I think I'll implement triton quant_atten and fused_mlp with lora support on this repo.
I finally tried the cuda quant attention by itself. fused_mlp doesn't help at all. On P6000, it is fastest inference, even vs auto_gptq with the same. Should have checked it sooner.
autograd/quant_atten
Output generated in 2.69 seconds (6.70 tokens/s, 18 tokens, context 66, seed 1725726905)
vs
AutoGPTQ
Output generated in 3.27 seconds (5.50 tokens/s, 18 tokens, context 66, seed 59570363)
Triton is consistently more slow on my ampere cards.
Some users on Reddit are reporting that a new version of GPTQ-for-LLaMA is providing better performance.
I wonder if this is better than the version used by this repo and compatible?