search

submission2019 / cnn-quantization

Quantization of Convolutional Neural networks.
237 stars 59 forks source link

can not reproduce the w4a4 quantization results based on the default commands #1

Closed brisker closed 5 years ago

brisker commented 5 years ago

After I run the following commands one by one: (they are all the default commands)

python inference/inference_sim.py -a resnet50 -b 256 -sm collect -ac --qtype int4

python inference/inference_sim.py -a resnet50 -b 256 -sm collect -ac --qtype int4 -pcq_a

python pytorch_quantizer/quantization/kmeans_quantization.py -a resnet50 -bits 4 -t quantize

python inference/inference_sim.py -a resnet50 -b 512 -sm use --qtype int4 -pcq_w -pcq_a -c laplace -qm 4 -qw f32

Then it gives me these lines and weird classification accuracy:

=> using pre-trained model 'resnet50'
Perform BN folding
Test: [0/98]    Time 24.867 (24.867)    Loss 9.3266 (9.3266)    Prec@1 0.586 (0.586)    Prec@5 2.539 (2.539)
Test: [10/98]   Time 1.649 (5.473)  Loss 9.9686 (10.5482)   Prec@1 0.781 (0.337)    Prec@5 3.906 (1.935)
Test: [20/98]   Time 16.422 (5.568) Loss 8.8218 (10.4333)   Prec@1 0.195 (0.456)    Prec@5 2.148 (1.990)
Test: [30/98]   Time 1.644 (5.145)  Loss 10.8783 (10.3225)  Prec@1 0.000 (0.372)    Prec@5 0.195 (1.928)
Test: [40/98]   Time 14.035 (5.262) Loss 10.0452 (10.3040)  Prec@1 0.781 (0.438)    Prec@5 3.711 (2.158)
Test: [50/98]   Time 1.646 (5.140)  Loss 9.2239 (10.1178)   Prec@1 0.391 (0.724)    Prec@5 1.758 (2.742)
Test: [60/98]   Time 15.470 (5.170) Loss 10.2627 (10.0315)  Prec@1 0.000 (0.832)    Prec@5 1.367 (3.099)
Test: [70/98]   Time 1.637 (5.067)  Loss 9.7789 (9.9481)    Prec@1 0.000 (0.935)    Prec@5 0.781 (3.381)
Test: [80/98]   Time 14.160 (5.121) Loss 9.5262 (9.8638)    Prec@1 1.758 (1.059)    Prec@5 7.031 (3.771)
Test: [90/98]   Time 1.667 (5.066)  Loss 9.0029 (9.8053)    Prec@1 0.781 (1.011)    Prec@5 4.297 (3.728)
 * Prec@1 1.066 Prec@5 4.016
ynahshan commented 5 years ago

Are you sure your path to imagenet dataset properly configured? You need to providing --data "PATH_TO_ILSVRC" or copy ILSVRC dir to ~/datasets/ILSVRC2012 Could you please run w/o quantization and check the result. python inference/inference_sim.py -a resnet50 -b 512

ynahshan commented 5 years ago

In addition you may need to run this script to prepare validation set for pytorch. https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh

You can find instructions here in pytorch documentation https://github.com/pytorch/examples/tree/master/imagenet

brisker commented 5 years ago

@ynahshan

root@debug-3r6heg347:/data/cnn-quantization-master# python inference/inference_sim.py -a resnet50 -b 512
=> using pre-trained model 'resnet50'
Perform BN folding
Test: [0/98]    Time 29.235 (29.235)    Loss 0.5774 (0.5774)    Prec@1 84.766 (84.766)  Prec@5 96.484 (96.484)
Test: [10/98]   Time 0.824 (8.053)  Loss 0.6856 (0.6809)    Prec@1 85.352 (83.043)  Prec@5 94.141 (95.455)
Test: [20/98]   Time 12.384 (6.726) Loss 0.6628 (0.6858)    Prec@1 82.812 (82.571)  Prec@5 96.094 (95.750)
Test: [30/98]   Time 0.824 (5.417)  Loss 0.8957 (0.7008)    Prec@1 78.125 (81.811)  Prec@5 94.727 (95.917)
Test: [40/98]   Time 9.252 (5.006)  Loss 1.4035 (0.7189)    Prec@1 65.430 (81.398)  Prec@5 88.867 (95.636)
Test: [50/98]   Time 0.824 (4.565)  Loss 1.0886 (0.8133)    Prec@1 71.289 (79.274)  Prec@5 91.797 (94.627)
Test: [60/98]   Time 10.787 (4.405) Loss 1.5274 (0.8610)    Prec@1 60.547 (78.253)  Prec@5 85.156 (94.003)
Test: [70/98]   Time 0.827 (4.136)  Loss 0.9446 (0.9043)    Prec@1 77.344 (77.308)  Prec@5 92.578 (93.566)
Test: [80/98]   Time 8.448 (4.071)  Loss 1.0883 (0.9397)    Prec@1 76.562 (76.654)  Prec@5 91.602 (93.065)
Test: [90/98]   Time 0.820 (3.930)  Loss 1.0987 (0.9689)    Prec@1 71.094 (75.998)  Prec@5 92.969 (92.793)
 * Prec@1 76.130 Prec@5 92.862
brisker commented 5 years ago

@ynahshan I think it is not the bug related to data.

ynahshan commented 5 years ago

I was able to reproduce the issue. The is issue related to kmeans model, I will debug it. Meanwhile you can run 4W4A mode w/o kmean. It should give you 69.8%. python inference/inference_sim.py -a resnet50 -b 512 -sm use --qtype int4 -pcq_w -pcq_a -c laplace -qw int4

brisker commented 5 years ago

@ynahshan In your 4w4a mode, what if no "per-channel-quantization" trick implemented on acticvations? What will the accuracy be? (Asking this because that lots of papers only implement per-channel on weights)

ynahshan commented 5 years ago

Per channel trick can be done on weights and activations independently. To check this you can run respectively. python inference/inference_sim.py -a resnet50 -b 512 -sm use --qtype int4 -c laplace -qw int4 -pcq_w python inference/inference_sim.py -a resnet50 -b 512 -sm use --qtype int4 -c laplace -qw int4 -pcq_a python inference/inference_sim.py -a resnet50 -b 512 -sm use --qtype int4 -c laplace -qw int4

brisker commented 5 years ago

@ynahshan

  1. What about the performance of uniform quantization, not non-uniform quantization? Is it simply 69.8% performance for resnet18-4w4a, by running: python inference/inference_sim.py -a resnet50 -b 512 -sm use --qtype int4 -pcq_w -pcq_a -c laplace -qw int4 ?
    1. Does this method support fine-tuning if training data is available?
    2. How to generate the claimed 72% top-1 accuracy results for resnet50-4a4w, after the code is debugged? What are the hyper-parameters?
    3. I tried to run without per-channel trick both on weights and activations, but the results are still very weird: 
      root@debug-3r6heg347:/data/cnn-quantization-master# python inference/inference_sim.py -a resnet50 -b 512 -sm use --qtype int4 -c laplace -qw int4
=> using pre-trained model 'resnet50'
Perform BN folding
Test: [0/98]    Time 19.385 (19.385)    Loss 14.5781 (14.5781)  Prec@1 0.000 (0.000)    Prec@5 0.000 (0.000)
Test: [10/98]   Time 1.398 (3.058)  Loss 13.2234 (12.1870)  Prec@1 0.000 (0.107)    Prec@5 0.000 (0.817)
Test: [20/98]   Time 4.273 (2.928)  Loss 14.1718 (13.0394)  Prec@1 0.000 (0.056)    Prec@5 0.000 (0.549)
Test: [30/98]   Time 5.744 (3.419)  Loss 10.3249 (13.0239)  Prec@1 0.000 (0.050)    Prec@5 2.344 (0.504)
Test: [40/98]   Time 1.435 (3.371)  Loss 11.8965 (12.8655)  Prec@1 0.000 (0.062)    Prec@5 0.000 (0.548)
Test: [50/98]   Time 1.389 (3.583)  Loss 11.5615 (12.5767)  Prec@1 0.000 (0.149)    Prec@5 0.000 (0.701)
Test: [60/98]   Time 1.394 (3.585)  Loss 10.3793 (12.4067)  Prec@1 0.000 (0.147)    Prec@5 0.195 (0.765)
Test: [70/98]   Time 1.394 (3.744)  Loss 12.1444 (12.2745)  Prec@1 0.000 (0.239)    Prec@5 0.000 (0.974)
Test: [80/98]   Time 1.397 (3.710)  Loss 11.1917 (12.1116)  Prec@1 0.195 (0.294)    Prec@5 1.367 (1.114)
Test: [90/98]   Time 1.391 (3.771)  Loss 13.7696 (12.0343)  Prec@1 0.000 (0.324)    Prec@5 0.000 (1.206)
* Prec@1 0.356 Prec@5 1.278050)    Prec@5 2.344 (0.504)
  2. What is the function of the following command , mentioned in your Readme file? It provides the quantized model with quantized weights, but not with quantized activations? Just for this? python pytorch_quantizer/quantization/kmeans_quantization.py -a resnet50 -bits 4 -t quantize
ynahshan commented 5 years ago

Fine-tuning is possible but not implemented in this code. W/o per-channel and 4W4A configuration results should be near 0 due to high quantization error.

brisker commented 5 years ago

@ynahshan Is the bug fixed?

Does this code support uniform quantization, not non-uniform quantization?

submission2019 commented 5 years ago

I pushed a fix. Please rerun python pytorch_quantizer/quantization/kmeans_quantization.py -a resnet18 -bits 4 -t quantize and python pytorch_quantizer/quantization/kmeans_quantization.py -a resnet50 -bits 4 -t quantize to regenerate kmeans quantized model weights. Than rerunning python inference/inference_sim.py -a resnet18 -b 512 -sm use --qtype int4 -pcq_w -pcq_a -c laplace -qm 4 -qw f32 and python inference/inference_sim.py -a resnet50 -b 512 -sm use --qtype int4 -pcq_w -pcq_a -c laplace -qm 4 -qw f32 Should give 65.2% and 72% accuracy respectively.

Uniform quantization is also supported by: python inference/inference_sim.py -a resnet50 -b 512 -sm use --qtype int4 -pcq_w -pcq_a -c laplace -qw int4

submission2019 commented 5 years ago

Fixed and closed.

brisker commented 5 years ago

In your paper , why is the weight quantization methods different from the activation quantization? Why not use the optimal clipping quantization methods which you use to quantize the activations, to quantize the weights? @ynahshan @submission2019

monkeyking commented 3 years ago

In addition you may need to run this script to prepare validation set for pytorch. https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh

You can find instructions here in pytorch documentation https://github.com/pytorch/examples/tree/master/imagenet

I met the same problem fix by using this method TY