search

facebookresearch / kill-the-bits

Code for: "And the bit goes down: Revisiting the quantization of neural networks"
Other
636 stars 123 forks source link

Reproducing memory of MaskRCNN and semisupervised ResNet-50 #39

Closed una-dinosauria closed 4 years ago

una-dinosauria commented 4 years ago

Hi @pierrestock,

Sorry to bother you again; there is one last aspect we've had trouble reproducing from the paper -- the memory taken by the MaskRCNN model, which is reported as 6.51 MB, with a 26x compression factor: image We also have a smaller discrepancy with the unsupervised resnet50 model, reported as 5.15 MB in the README: image

We have followed the paper, and counted all the codebooks as being stored in float16 format: image

However, the paper does not say which encoding is used by the bnorm layers or other layers ignored for the purpose of compression. To reproduce the results on Imagenet, we have used float32 encoding for these two cases. This gives us the following results:

resnet50_small_blocks.pth
bits: 42705152
Byts: 5338144.0
  KB: 5213.03125
  MB: 5.09

resnet50_large_blocks.pth
bits: 26710272
Byts: 3338784.0
  KB: 3260.53125
  MB: 3.18

resnet18_small_blocks.pth
bits: 12927232
Byts: 1615904.0
  KB: 1578.03125
  MB: 1.54

resnet50_semisup_small_blocks.pth
bits: 43655424
Byts: 5456928.0
  KB: 5329.03125
  MB: 5.20

resnet18_large_blocks.pth
bits: 8634624
Byts: 1079328.0
  KB: 1054.03125
  MB: 1.03

mask_r_cnn.pth
bits: 55743008
Byts: 6967876.0
  KB: 6804.56640625
  MB: 6.65

There are however, two small discrepances:

  1. the semi-supervised model, reported at 5.15 MB while we get 5.20 MB, and
  2. the maskrcnn model, reported at 6.51 MB while we get 6.65 MB -- this matches the number that we report in our paper.

We have tried counting the uncompressed and bnorm layers as float16, but that also gives different results than those reported in the paper:

half_codebooks: True, half_weights: True
mask_r_cnn.pth
bits: 54209296
Byts: 6776162.0
  KB: 6617.345703125
  MB: 6.46

with similar results for the semi-supervised resnet50.

half_codebooks: True, half_weights: True
resnet50_semisup_small_blocks.pth
bits: 42114688
Byts: 5264336.0
  KB: 5140.953125
  MB: 5.02

So, my question is: could you please explain how you obtained the model size for mask-rcnn and the unsupervised resnet50?

I have put together a gist that makes it easy to see how we computed our numbers: https://gist.github.com/una-dinosauria/e528b91de3ca9ab108cbf00aba3d9c2a. Please do make sure to run this on the mask_r_cnn branch of the codebase, as the mask_r_cnn.pth model is missing all the compressed biases on master.

Thank you in advance, Julieta

pierrestock commented 4 years ago

Hi Julieta,

Thanks a lot for your thorough analysis, which is very helpful. Let me answer to your questions below:

In our next paper, we use int8 to store the non-quantized layers, and store the centroids in int8 but the important thing is to compare apples to apples. Unfortunately, I must have forgotten to count some biases for both the semi-supervised and Mask R-CNN networks and I agree with your calculations. I will correct the paper (together with the previous issue #38).

Thanks you again for your interest and your work, looking forward to see your future research!

Pierre

una-dinosauria commented 4 years ago

Thanks again for your fast response, Pierre.

I have created a small PR to change the README to match the updated size of the semisupervised ResNet-50 over here: https://github.com/facebookresearch/kill-the-bits/pull/40

Regarding the Mask-RCNN part, would you mind keeping this issue open until the paper is updated?

Cheers,

pierrestock commented 4 years ago

Thanks Julieta. Of course I can keep the issue open until the paper is updated and I will ping you when it's done.

Wishing you good luck for the ICLR incoming reviews (if any) and for all future papers!

Pierre

pierrestock commented 4 years ago

Hi Julieta,

Paper has been updated, see here as well as readme for Mask R-CNN here.

Thanks, Pierre