Large performance drop after quantizing my own depthwise model

[muskedunder]

I followed your example auto_test with my own depthwise deparable CNN. After a few epochs of training my Keras model has an accuracy of 98.12% on the MNIST test set. After quantization the NNoM model has an accuracy of 12.95%. I do expect some performance drop but this is such a large drop that I rather think it is a bug. Here is the model summary:

Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 28, 28, 1)         0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 26, 26, 12)        120       
_________________________________________________________________
batch_normalization_1 (Batch (None, 26, 26, 12)        48        
_________________________________________________________________
activation_1 (Activation)    (None, 26, 26, 12)        0         
_________________________________________________________________
depthwise_conv2d_1 (Depthwis (None, 24, 24, 12)        120       
_________________________________________________________________
batch_normalization_2 (Batch (None, 24, 24, 12)        48        
_________________________________________________________________
activation_2 (Activation)    (None, 24, 24, 12)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 24, 24, 16)        208       
_________________________________________________________________
batch_normalization_3 (Batch (None, 24, 24, 16)        64        
_________________________________________________________________
activation_3 (Activation)    (None, 24, 24, 16)        0         
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 12, 12, 16)        0         
_________________________________________________________________
depthwise_conv2d_2 (Depthwis (None, 10, 10, 16)        160       
_________________________________________________________________
batch_normalization_4 (Batch (None, 10, 10, 16)        64        
_________________________________________________________________
activation_4 (Activation)    (None, 10, 10, 16)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 10, 10, 32)        544       
_________________________________________________________________
batch_normalization_5 (Batch (None, 10, 10, 32)        128       
_________________________________________________________________
activation_5 (Activation)    (None, 10, 10, 32)        0         
_________________________________________________________________
depthwise_conv2d_3 (Depthwis (None, 8, 8, 32)          320       
_________________________________________________________________
batch_normalization_6 (Batch (None, 8, 8, 32)          128       
_________________________________________________________________
activation_6 (Activation)    (None, 8, 8, 32)          0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 8, 8, 64)          2112      
_________________________________________________________________
batch_normalization_7 (Batch (None, 8, 8, 64)          256       
_________________________________________________________________
activation_7 (Activation)    (None, 8, 8, 64)          0         
_________________________________________________________________
depthwise_conv2d_4 (Depthwis (None, 6, 6, 64)          640       
_________________________________________________________________
batch_normalization_8 (Batch (None, 6, 6, 64)          256       
_________________________________________________________________
activation_8 (Activation)    (None, 6, 6, 64)          0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 6, 6, 96)          6240      
_________________________________________________________________
batch_normalization_9 (Batch (None, 6, 6, 96)          384       
_________________________________________________________________
activation_9 (Activation)    (None, 6, 6, 96)          0         
_________________________________________________________________
global_max_pooling2d_1 (Glob (None, 96)                0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 96)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 96)                9312      
_________________________________________________________________
re_lu_1 (ReLU)               (None, 96)                0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 96)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 10)                970       
_________________________________________________________________
activation_10 (Activation)   (None, 10)                0         
=================================================================
Total params: 22,122
Trainable params: 21,434
Non-trainable params: 688

---

[muskedunder]

I think I found the source of error. If I replace the global-maxpool-layer with a regular maxpool-layer (with same poolsize as the global-maxpool-layer) plus a flatten-layer, it works as expected!

[majianjia]

Hi, I think it might be a bug, but I can't replicate this issue. Both max pooling and Global max pooling are using the same max_pooling backend. So, it could be something wrong in setting up the configuration in global pooling. Would you might to provide your model for me to investigate a bit more?

[muskedunder]

Sure, what is the best way to share a .h5-file with you?

Edit: You can find it here https://github.com/ColinNordin/depthwise-mnist-model