Problems with memory usage and model loading

[Forival]

I used the ”fake quant“ code you gave to perform llama quantification. During the model loading stage, I found that all the parameters loaded were of the fp16, and the memory usage was almost the same as the original llama model. May I ask if I use real quant parameters to load? how to get the extremely low memory use in the paper? How to correctly load a quantized model (taking 4bit as an example)？

[ChenMnZ]

If you want to obtain practical memory reduction and speedup, you should leverage mlc-llm, refer https://github.com/OpenGVLab/OmniQuant/blob/main/runing_quantized_models_with_mlc_llm.ipynb for more details.