Open pineking opened 1 year ago
does someone have compared the inference speed of 4bit quantized model with the origin FP16 model? is it faster than the origin FP16 model?
I tested, but int4 costs 2 time of FP16. Anything wrong?
same,do you know why?
does someone have compared the inference speed of 4bit quantized model with the origin FP16 model? is it faster than the origin FP16 model?