AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:
1.59k
stars
186
forks
source link
is there a reason why gemm stores the tensors reversed (in, out) vs (out, in) ? #234
Closed
vince62s closed 9 months ago
it makes things difficult when we want to handle tensor parallelism .....