Quantization seems to be doesn't produce good accuracy ?

[padeirocarlos]

So I tried running your code on my dataset with a pre-trained ResNet50 model. I got these results

Full precision model i got accuracy of : MobileNetV2 (58.19) Quantized model (W8A8) i got accuracy of : MobileNetV2 (12.02) Quantized model (W6A6) i got accuracy of : MobileNetV2 (10.12)

Full precision model i got accuracy of : ResNet-50 (65.16) Quantized model (W8A8) i got accuracy of : ResNet-50 (13.22) Quantized model (W6A6) i got accuracy of : ResNet-50 (11.02)

My accuracy however does not come nearly as close to the float model which is around 58.19% and 65.16% but after quantization Are there additional settings I missed?

[BobxmuMa]

To use MRECG on your dataset, you need to swap the calibration dataset from ImageNet to your dataset.

If quantization accuracy cannot be restored, can you provide specific quantization configurations.

[padeirocarlos]

Do you have any more information about your experiment configuration?

a) I first quantized the ResNet50 and MobileNetV2 and I test it on classification it did not work. Then I use this quantized model as backbone for detection (Faster-RCNN) even though it did not work.

b) I quantized the first layer and block with 8bit;

c) I am running reconstruction block with 20000 iteration;

How does the loss change during training?
The loss during training is very high. It vary from 200.000 to 400.000 ; Which make the training not convert

Did you use your dataset as calibration data?
Yes! I use my dataset as calibration data. I separate the dataset half for training and half for calibration data.

Is there any other setting?

[BobxmuMa]

The variation in loss does not seem to be normal. Please try to make the loss converge to 0.

On other datasets, you can try to adjust the self.weight parameter to balance the rounding loss and rec loss.

[padeirocarlos]

This will not change the main idea of Post-Training Quantization which is not to change or train the original weight? if we adjust the self.weight parameter to balance the rounding loss and rec loss. Sorry, for this question may seems to be a dummy question but is good the clarify!

Did you try this approach in this paper? I cant see it on this code!

[BobxmuMa]

This will not change the main idea of Post-Training Quantization which is not to change or train the original weight? if we adjust the self.weight parameter to balance the rounding loss and rec loss. Sorry, for this question may seems to be a dummy question but is good the clarify!

Did you try this approach in this paper? I cant see it on this code!

I mean adjusting the following hyperparameter self.weight. this hyperparameter serves to balance rec loss and rounding loss. self.weight does not refer to the original model weights. Please refer to L225 in MRECG.py.

                    round_loss += self.weight * (1 - ((round_vals - .5).abs() * 2).pow(b)).sum()

[padeirocarlos]

I see. It is clear now!

I thought it was original model weights. I found this scenario in my dataset where rec loss is very bigger than rounding loss! So will try to use this hyper-parameter to balance rec loss and rounding! unfortunately it as default value as 1. Which was best value in ImageNet dataset?

[BobxmuMa]

I see. It is clear now!

I thought it was original model weights. I found this scenario in my dataset where rec loss is very bigger than rounding loss! So will try to use this hyper-parameter to balance rec loss and rounding! unfortunately it as default value as 1. Which was best value in ImageNet dataset?

The exact value depends on the dataset and the model. You can refer to the ratio of rec loss to rounding loss in MRECG.