Kaihui-intel
commented
1 month ago Hello, @zhangxu223 Thanks for your interest in Intel(R) Neural Compressor.
For 1, The model has been successfully quantized, you should use q_model.save("saved_results") to save the quantized model and config. The model size of the best_model.pt file in the folder saved_results is approximately one fourth of the original model.
You can refer to the document PTQ.
For 2, you can set op_name_dict and op_type_dict in config class to achieve the purpose.
# set op_type_dict
op_type_dict = {"Conv": {"weight": {"dtype": ["fp32"]}, "activation": {"dtype": ["fp32"]}}}
conf = PostTrainingQuantConfig(op_type_dict=op_type_dict)
# or set op_name_dict
op_name_dict = {
"layer1.0.conv1": {
"activation": {
"dtype": ["fp32"],
},
"weight": {
"dtype": ["fp32"],
},
}
}
conf = PostTrainingQuantConfig(op_name_dict=op_name_dict)You can refer to the document specify-quantization-rules.
I hope the above information is useful to you, and I am looking forward to your reply.
zhangxu223
I've encountered an issue where the quantized model size is twice as large as the original model, which contradicts the expected result of reducing the model size after quantization.
Original Model Size: 130.96 MB Quantized Model Size: Approximately 261.92 MB (double the original size)
This is my code:
This is my logs:
I have two questions:
When using Neural Compressor for quantization, I noticed that the size of the quantized model is larger than the original one. I expected the model size to decrease after quantization, but it actually increased. The logs indicate that the model was successfully quantized to int8. Is this behavior normal? Did I successfully quantize the model? Additionally, my FP32 model was initially trained on a GPU, but this post-training quantization (PTQ) was performed on a CPU. Could this be related to the increase in model size?
Is there a way to inspect the precision of each layer in the quantized model? I would like to verify the precision (e.g., FP32, INT8) at which each layer is operating to better understand the impact of the quantization process.