I used your real_quant parameters to obtain the quantized llama7b model and tested the inference speed on A10. However, for both w4a16 and w4a4, the inference speed was only 7 tokens/s, and it also exceeded 7GB in terms of memory usage. These results differ significantly from the reported results in the paper(100+tokens\s, 5.7GB). Is it possible for you to open-source your testing code?
I used your real_quant parameters to obtain the quantized llama7b model and tested the inference speed on A10. However, for both w4a16 and w4a4, the inference speed was only 7 tokens/s, and it also exceeded 7GB in terms of memory usage. These results differ significantly from the reported results in the paper(100+tokens\s, 5.7GB). Is it possible for you to open-source your testing code?