I used the ”fake quant“ code you gave to perform llama quantification. During the model loading stage, I found that all the parameters loaded were of the fp16, and the memory usage was almost the same as the original llama model. May I ask if I use real quant parameters to load? how to get the extremely low memory use in the paper? How to correctly load a quantized model (taking 4bit as an example)?
I used the ”fake quant“ code you gave to perform llama quantification. During the model loading stage, I found that all the parameters loaded were of the fp16, and the memory usage was almost the same as the original llama model. May I ask if I use real quant parameters to load? how to get the extremely low memory use in the paper? How to correctly load a quantized model (taking 4bit as an example)?