search

bytedance / MRECG

Apache License 2.0
35 stars 4 forks source link

mobileNetV2 setting problem, the corresponding accuracy is not reached #10

Open ChengShaoFong opened 4 months ago

ChengShaoFong commented 4 months ago

Hi, I'm trying to apply BRECQ settings to achieve the accuracy of mobileNetV2. Prior to this, resnet18/50 was close to the accuracy given in the paper, but there was a big difference on mobileNetV2. Do you know why? I tried using AdaRound with the weight set to 0.1, and everything else was set as per the original code.

BobxmuMa commented 4 months ago

Sorry for the late reply. Hope you can answer some of my questions so that I can offer you specific advice.

What is your experimental setup for MobileNetV2, such as what kind of quantization bit-width, how large the MobileNetV2 model is? How does the accuracy differ, and what is the accuracy you have reproduced? Also, what is the final loss performance, including rec_loss and rounding loss?

ChengShaoFong commented 4 months ago

Glad you can reply

The pre-trained model I used was downloaded from https://github.com/yhhhli/BRECQ/releases/tag/v1.0, and the full test accuracy was 72.632

Apply block reconstruction ,weight use per channel & act use per layer ; args.weight=0.1; batch=32; p=2.4; iter=2w; lr=4e-5 The 8-bit settings before and after use BRECQ configuration, and the act of the first layer and the penultimate layer are set to 8-bit. The precision is: w4a32: 71.972 w4a4= 62.169 w4a4: 63.406

The interesting point is that if I don't process the shortcut layer into account, the accuracy reaches 67.8, so I wonder if others will take it into account when comparing data.

Here is the loss of each block of my act :

Ignore reconstruction of layer 0 Reconstruction for block 1 Total loss: 48.682 (rec:48.682, round:0.000) b=0.00 count=500 Total loss: 50.978 (rec:50.978, round:0.000) b=0.00 count=1000 Total loss: 48.023 (rec:48.023, round:0.000) b=0.00 count=1500 Total loss: 50.625 (rec:50.625, round:0.000) b=0.00 count=2000 Total loss: 46.462 (rec:46.462, round:0.000) b=0.00 count=2500 Total loss: 46.145 (rec:46.145, round:0.000) b=0.00 count=3000 Total loss: 52.556 (rec:52.556, round:0.000) b=0.00 count=3500 Total loss: 49.150 (rec:49.150, round:0.000) b=0.00 count=4000 Total loss: 46.671 (rec:46.671, round:0.000) b=0.00 count=4500 Total loss: 44.437 (rec:44.437, round:0.000) b=0.00 count=5000 Reconstruction for block 2 Total loss: 40.665 (rec:40.665, round:0.000) b=0.00 count=500 Total loss: 41.384 (rec:41.384, round:0.000) b=0.00 count=1000 Total loss: 43.291 (rec:43.291, round:0.000) b=0.00 count=1500 Total loss: 40.566 (rec:40.566, round:0.000) b=0.00 count=2000 Total loss: 41.873 (rec:41.873, round:0.000) b=0.00 count=2500 Total loss: 40.336 (rec:40.336, round:0.000) b=0.00 count=3000 Total loss: 39.902 (rec:39.902, round:0.000) b=0.00 count=3500 Total loss: 40.370 (rec:40.370, round:0.000) b=0.00 count=4000 Total loss: 41.219 (rec:41.219, round:0.000) b=0.00 count=4500 Total loss: 43.364 (rec:43.364, round:0.000) b=0.00 count=5000 Reconstruction for block 3

Total loss: 24.624 (rec:24.624, round:0.000) b=0.00 count=500 Total loss: 25.764 (rec:25.764, round:0.000) b=0.00 count=1000 Total loss: 24.476 (rec:24.476, round:0.000) b=0.00 count=1500 Total loss: 26.010 (rec:26.010, round:0.000) b=0.00 count=2000 Total loss: 24.139 (rec:24.139, round:0.000) b=0.00 count=2500 Total loss: 24.031 (rec:24.031, round:0.000) b=0.00 count=3000 Total loss: 24.570 (rec:24.570, round:0.000) b=0.00 count=3500 Total loss: 24.862 (rec:24.862, round:0.000) b=0.00 count=4000 Total loss: 24.382 (rec:24.382, round:0.000) b=0.00 count=4500 Total loss: 24.493 (rec:24.493, round:0.000) b=0.00 count=5000 Reconstruction for block 4 Total loss: 14.370 (rec:14.370, round:0.000) b=0.00 count=500 Total loss: 13.380 (rec:13.380, round:0.000) b=0.00 count=1000 Total loss: 12.341 (rec:12.341, round:0.000) b=0.00 count=1500 Total loss: 15.856 (rec:15.856, round:0.000) b=0.00 count=2000 Total loss: 13.015 (rec:13.015, round:0.000) b=0.00 count=2500 Total loss: 13.668 (rec:13.668, round:0.000) b=0.00 count=3000 Total loss: 13.035 (rec:13.035, round:0.000) b=0.00 count=3500 Total loss: 12.802 (rec:12.802, round:0.000) b=0.00 count=4000 Total loss: 11.681 (rec:11.681, round:0.000) b=0.00 count=4500 Total loss: 13.462 (rec:13.462, round:0.000) b=0.00 count=5000 Reconstruction for block 5 Total loss: 15.440 (rec:15.440, round:0.000) b=0.00 count=500 Total loss: 13.151 (rec:13.151, round:0.000) b=0.00 count=1000 Total loss: 14.989 (rec:14.989, round:0.000) b=0.00 count=1500 Total loss: 15.174 (rec:15.174, round:0.000) b=0.00 count=2000 Total loss: 14.988 (rec:14.988, round:0.000) b=0.00 count=2500 Total loss: 13.612 (rec:13.612, round:0.000) b=0.00 count=3000 Total loss: 14.517 (rec:14.517, round:0.000) b=0.00 count=3500 Total loss: 15.658 (rec:15.658, round:0.000) b=0.00 count=4000 Total loss: 14.801 (rec:14.801, round:0.000) b=0.00 count=4500 Total loss: 13.478 (rec:13.478, round:0.000) b=0.00 count=5000 Reconstruction for block 6 Total loss: 19.882 (rec:19.882, round:0.000) b=0.00 count=500 Total loss: 20.726 (rec:20.726, round:0.000) b=0.00 count=1000 Total loss: 17.074 (rec:17.074, round:0.000) b=0.00 count=1500 Total loss: 15.269 (rec:15.269, round:0.000) b=0.00 count=2000 Total loss: 18.087 (rec:18.087, round:0.000) b=0.00 count=2500 Total loss: 17.721 (rec:17.721, round:0.000) b=0.00 count=3000 Total loss: 16.936 (rec:16.936, round:0.000) b=0.00 count=3500 Total loss: 19.266 (rec:19.266, round:0.000) b=0.00 count=4000 Total loss: 20.506 (rec:20.506, round:0.000) b=0.00 count=4500 Total loss: 17.911 (rec:17.911, round:0.000) b=0.00 count=5000 Reconstruction for block 7 Total loss: 20.461 (rec:20.461, round:0.000) b=0.00 count=500 Total loss: 19.785 (rec:19.785, round:0.000) b=0.00 count=1000 Total loss: 18.110 (rec:18.110, round:0.000) b=0.00 count=1500 Total loss: 20.168 (rec:20.168, round:0.000) b=0.00 count=2000 Total loss: 18.743 (rec:18.743, round:0.000) b=0.00 count=2500 Total loss: 19.992 (rec:19.992, round:0.000) b=0.00 count=3000 Total loss: 19.822 (rec:19.822, round:0.000) b=0.00 count=3500 Total loss: 19.842 (rec:19.842, round:0.000) b=0.00 count=4000 Total loss: 18.977 (rec:18.977, round:0.000) b=0.00 count=4500 Total loss: 19.915 (rec:19.915, round:0.000) b=0.00 count=5000 Reconstruction for block 8 Total loss: 18.518 (rec:18.518, round:0.000) b=0.00 count=500 Total loss: 16.812 (rec:16.812, round:0.000) b=0.00 count=1000 Total loss: 17.153 (rec:17.153, round:0.000) b=0.00 count=1500 Total loss: 15.076 (rec:15.076, round:0.000) b=0.00 count=2000 Total loss: 16.035 (rec:16.035, round:0.000) b=0.00 count=2500 Total loss: 16.376 (rec:16.376, round:0.000) b=0.00 count=3000 Total loss: 15.831 (rec:15.831, round:0.000) b=0.00 count=3500 Total loss: 17.081 (rec:17.081, round:0.000) b=0.00 count=4000 Total loss: 17.177 (rec:17.177, round:0.000) b=0.00 count=4500 Total loss: 18.839 (rec:18.839, round:0.000) b=0.00 count=5000 Reconstruction for block 9 Total loss: 19.684 (rec:19.684, round:0.000) b=0.00 count=500 Total loss: 17.595 (rec:17.595, round:0.000) b=0.00 count=1000 Total loss: 20.284 (rec:20.284, round:0.000) b=0.00 count=1500 Total loss: 15.616 (rec:15.616, round:0.000) b=0.00 count=2000 Total loss: 16.382 (rec:16.382, round:0.000) b=0.00 count=2500 Total loss: 18.332 (rec:18.332, round:0.000) b=0.00 count=3000 Total loss: 18.790 (rec:18.790, round:0.000) b=0.00 count=3500 Total loss: 14.989 (rec:14.989, round:0.000) b=0.00 count=4000 Total loss: 17.534 (rec:17.534, round:0.000) b=0.00 count=4500 Total loss: 16.784 (rec:16.784, round:0.000) b=0.00 count=5000 Reconstruction for block 10 Total loss: 17.994 (rec:17.994, round:0.000) b=0.00 count=500 Total loss: 19.332 (rec:19.332, round:0.000) b=0.00 count=1000 Total loss: 17.317 (rec:17.317, round:0.000) b=0.00 count=1500 Total loss: 18.364 (rec:18.364, round:0.000) b=0.00 count=2000 Total loss: 17.364 (rec:17.364, round:0.000) b=0.00 count=2500 Total loss: 17.261 (rec:17.261, round:0.000) b=0.00 count=3000 Total loss: 18.790 (rec:18.790, round:0.000) b=0.00 count=3500 Total loss: 16.771 (rec:16.771, round:0.000) b=0.00 count=4000 Total loss: 16.710 (rec:16.710, round:0.000) b=0.00 count=4500 Total loss: 16.103 (rec:16.103, round:0.000) b=0.00 count=5000 Reconstruction for block 11 Total loss: 20.854 (rec:20.854, round:0.000) b=0.00 count=500 Total loss: 19.935 (rec:19.935, round:0.000) b=0.00 count=1000 Total loss: 20.100 (rec:20.100, round:0.000) b=0.00 count=1500 Total loss: 21.607 (rec:21.607, round:0.000) b=0.00 count=2000 Total loss: 19.258 (rec:19.258, round:0.000) b=0.00 count=2500 Total loss: 20.736 (rec:20.736, round:0.000) b=0.00 count=3000 Total loss: 20.494 (rec:20.494, round:0.000) b=0.00 count=3500 Total loss: 19.474 (rec:19.474, round:0.000) b=0.00 count=4000 Total loss: 19.464 (rec:19.464, round:0.000) b=0.00 count=4500 Total loss: 19.889 (rec:19.889, round:0.000) b=0.00 count=5000 Reconstruction for block 12 Total loss: 21.663 (rec:21.663, round:0.000) b=0.00 count=500 Total loss: 26.878 (rec:26.878, round:0.000) b=0.00 count=1000 Total loss: 23.160 (rec:23.160, round:0.000) b=0.00 count=1500 Total loss: 21.607 (rec:21.607, round:0.000) b=0.00 count=2000 Total loss: 22.410 (rec:22.410, round:0.000) b=0.00 count=2500 Total loss: 20.745 (rec:20.745, round:0.000) b=0.00 count=3000 Total loss: 22.072 (rec:22.072, round:0.000) b=0.00 count=3500 Total loss: 20.519 (rec:20.519, round:0.000) b=0.00 count=4000 Total loss: 22.557 (rec:22.557, round:0.000) b=0.00 count=4500 Total loss: 20.512 (rec:20.512, round:0.000) b=0.00 count=5000 Reconstruction for block 13 Total loss: 33.823 (rec:33.823, round:0.000) b=0.00 count=500 Total loss: 33.683 (rec:33.683, round:0.000) b=0.00 count=1000 Total loss: 38.940 (rec:38.940, round:0.000) b=0.00 count=1500 Total loss: 31.121 (rec:31.121, round:0.000) b=0.00 count=2000 Total loss: 32.924 (rec:32.924, round:0.000) b=0.00 count=2500 Total loss: 32.409 (rec:32.409, round:0.000) b=0.00 count=3000 Total loss: 29.779 (rec:29.779, round:0.000) b=0.00 count=3500 Total loss: 27.686 (rec:27.686, round:0.000) b=0.00 count=4000 Total loss: 28.597 (rec:28.597, round:0.000) b=0.00 count=4500 Total loss: 29.688 (rec:29.688, round:0.000) b=0.00 count=5000 Reconstruction for block 14 Total loss: 30.750 (rec:30.750, round:0.000) b=0.00 count=500 Total loss: 22.425 (rec:22.425, round:0.000) b=0.00 count=1000 Total loss: 24.797 (rec:24.797, round:0.000) b=0.00 count=1500 Total loss: 30.494 (rec:30.494, round:0.000) b=0.00 count=2000 Total loss: 22.433 (rec:22.433, round:0.000) b=0.00 count=2500 Total loss: 25.334 (rec:25.334, round:0.000) b=0.00 count=3000 Total loss: 23.248 (rec:23.248, round:0.000) b=0.00 count=3500 Total loss: 28.453 (rec:28.453, round:0.000) b=0.00 count=4000 Total loss: 25.327 (rec:25.327, round:0.000) b=0.00 count=4500 Total loss: 25.764 (rec:25.764, round:0.000) b=0.00 count=5000 Reconstruction for block 15 Total loss: 41.723 (rec:41.723, round:0.000) b=0.00 count=500 Total loss: 38.562 (rec:38.562, round:0.000) b=0.00 count=1000 Total loss: 33.900 (rec:33.900, round:0.000) b=0.00 count=1500 Total loss: 41.896 (rec:41.896, round:0.000) b=0.00 count=2000 Total loss: 33.368 (rec:33.368, round:0.000) b=0.00 count=2500 Total loss: 28.529 (rec:28.529, round:0.000) b=0.00 count=3000 Total loss: 48.753 (rec:48.753, round:0.000) b=0.00 count=3500 Total loss: 35.288 (rec:35.288, round:0.000) b=0.00 count=4000 Total loss: 42.617 (rec:42.617, round:0.000) b=0.00 count=4500 Total loss: 48.335 (rec:48.335, round:0.000) b=0.00 count=5000 Reconstruction for block 16 Total loss: 90.652 (rec:90.652, round:0.000) b=0.00 count=500 Total loss: 81.390 (rec:81.390, round:0.000) b=0.00 count=1000 Total loss: 86.990 (rec:86.990, round:0.000) b=0.00 count=1500 Total loss: 85.142 (rec:85.142, round:0.000) b=0.00 count=2000 Total loss: 167.691 (rec:167.691, round:0.000) b=0.00 count=2500 Total loss: 82.074 (rec:82.074, round:0.000) b=0.00 count=3000 Total loss: 118.604 (rec:118.604, round:0.000) b=0.00 count=3500 Total loss: 85.894 (rec:85.894, round:0.000) b=0.00 count=4000 Total loss: 124.612 (rec:124.612, round:0.000) b=0.00 count=4500 Total loss: 137.315 (rec:137.315, round:0.000) b=0.00 count=5000 Reconstruction for block 17 Total loss: 39.041 (rec:39.041, round:0.000) b=0.00 count=500 Total loss: 46.477 (rec:46.477, round:0.000) b=0.00 count=1000 Total loss: 41.076 (rec:41.076, round:0.000) b=0.00 count=1500 Total loss: 38.570 (rec:38.570, round:0.000) b=0.00 count=2000 Total loss: 40.120 (rec:40.120, round:0.000) b=0.00 count=2500 Total loss: 38.874 (rec:38.874, round:0.000) b=0.00 count=3000 Total loss: 37.237 (rec:37.237, round:0.000) b=0.00 count=3500 Total loss: 40.781 (rec:40.781, round:0.000) b=0.00 count=4000 Total loss: 41.554 (rec:41.554, round:0.000) b=0.00 count=4500 Total loss: 39.810 (rec:39.810, round:0.000) b=0.00 count=5000

Reconstruction for layer 0 Total loss: 148.391 (rec:148.391, round:0.000) b=0.00 count=500 Total loss: 140.079 (rec:140.079, round:0.000) b=0.00 count=1000 Total loss: 145.885 (rec:145.885, round:0.000) b=0.00 count=1500 Total loss: 140.514 (rec:140.514, round:0.000) b=0.00 count=2000 Total loss: 144.596 (rec:144.596, round:0.000) b=0.00 count=2500 Total loss: 146.085 (rec:146.085, round:0.000) b=0.00 count=3000 Total loss: 151.463 (rec:151.463, round:0.000) b=0.00 count=3500 Total loss: 139.211 (rec:139.211, round:0.000) b=0.00 count=4000 Total loss: 144.337 (rec:144.337, round:0.000) b=0.00 count=4500 Total loss: 132.320 (rec:132.320, round:0.000) b=0.00 count=5000

Reconstruction for layer 1 Total loss: 0.580 (rec:0.580, round:0.000) b=0.00 count=500 Total loss: 0.660 (rec:0.660, round:0.000) b=0.00 count=1000 Total loss: 0.779 (rec:0.779, round:0.000) b=0.00 count=1500 Total loss: 0.695 (rec:0.695, round:0.000) b=0.00 count=2000 Total loss: 0.697 (rec:0.697, round:0.000) b=0.00 count=2500 Total loss: 0.676 (rec:0.676, round:0.000) b=0.00 count=3000 Total loss: 0.688 (rec:0.688, round:0.000) b=0.00 count=3500 Total loss: 0.721 (rec:0.721, round:0.000) b=0.00 count=4000 Total loss: 0.576 (rec:0.576, round:0.000) b=0.00 count=4500 Total loss: 0.723 (rec:0.723, round:0.000) b=0.00 count=5000 Weight and Activation after Acc: Test: [ 0/1563] Time 0.332 ( 0.332) Acc@1 93.75 ( 93.75) Acc@5 96.88 ( 96.88) Test: [ 100/1563] Time 0.074 ( 0.078) Acc@1 59.38 ( 70.98) Acc@5 78.12 ( 90.47) Test: [ 200/1563] Time 0.077 ( 0.077) Acc@1 68.75 ( 71.92) Acc@5 84.38 ( 90.36) Test: [ 300/1563] Time 0.074 ( 0.076) Acc@1 68.75 ( 71.64) Acc@5 93.75 ( 90.53) Test: [ 400/1563] Time 0.077 ( 0.076) Acc@1 62.50 ( 71.11) Acc@5 81.25 ( 90.79) Test: [ 500/1563] Time 0.077 ( 0.076) Acc@1 71.88 ( 70.99) Acc@5 96.88 ( 91.00) Test: [ 600/1563] Time 0.074 ( 0.076) Acc@1 87.50 ( 71.43) Acc@5 93.75 ( 90.99) Test: [ 700/1563] Time 0.074 ( 0.076) Acc@1 81.25 ( 69.75) Acc@5 90.62 ( 89.79) Test: [ 800/1563] Time 0.074 ( 0.076) Acc@1 53.12 ( 67.96) Acc@5 84.38 ( 88.42) Test: [ 900/1563] Time 0.080 ( 0.076) Acc@1 84.38 ( 67.16) Acc@5 90.62 ( 87.64) Test: [1000/1563] Time 0.076 ( 0.076) Acc@1 87.50 ( 66.01) Acc@5 100.00 ( 86.72) Test: [1100/1563] Time 0.076 ( 0.075) Acc@1 71.88 ( 65.20) Acc@5 84.38 ( 86.07) Test: [1200/1563] Time 0.076 ( 0.075) Acc@1 75.00 ( 64.37) Acc@5 87.50 ( 85.43) Test: [1300/1563] Time 0.076 ( 0.075) Acc@1 78.12 ( 63.91) Acc@5 84.38 ( 84.98) Test: [1400/1563] Time 0.076 ( 0.075) Acc@1 53.12 ( 63.34) Acc@5 75.00 ( 84.45) Test: [1500/1563] Time 0.074 ( 0.075) Acc@1 37.50 ( 63.31) Acc@5 81.25 ( 84.48)

BobxmuMa commented 4 months ago
  1. During reconstruction, we generally consider the output after the shortcut layers.
  2. You can increase the batch_size to 256.
  3. Please ensure that the corresponding modules have been jointly optimized.

Following the above settings, the accuracy of MRECG can usually be reproduced.

ChengShaoFong commented 4 months ago

Is it only the recon batch or is the cali batch also needed? I only have one graphics card and it cannot be opened to 256. Before that I tried 64batch for recon but the effect was not significantly improved or even worsened. I would like to ask if you have any other suggestions?

Also, regarding the act_quantizer that processes the shortcut layer, where do you put it? As there is no ReLU6 at the end of the block and the act_quantizer is done after the addition, I think there are more decays there.

BobxmuMa commented 4 months ago

Is it only the recon batch or is the cali batch also needed? I only have one graphics card and it cannot be opened to 256. Before that I tried 64batch for recon but the effect was not significantly improved or even worsened. I would like to ask if you have any other suggestions?

Also, regarding the act_quantizer that processes the shortcut layer, where do you put it? As there is no ReLU6 at the end of the block and the act_quantizer is done after the addition, I think there are more decays there.

We ignore act_quantizer after addition, as our experiments indicate that its inclusion significantly reduces precision. Thus, our block reconstruction focuses solely on the segment preceding the addition operation. And the input of the next block takes the output of the act_quantizer after addition.