search

Eric-mingjie / rethinking-network-pruning

Rethinking the Value of Network Pruning (Pytorch) (ICLR 2019)
MIT License
1.5k stars 295 forks source link

A question about the pruning algorithm #18

Open zhoudaquan opened 5 years ago

zhoudaquan commented 5 years ago

Dear author,

I just found an issue that with the current algorithm, the threshold for the pruning is selected based on the batch norm scaling factors of all layers. Hence, it is possible that all the scaling factors of certain layer are below the threshold and hence all channels in that layer is masked. In such cases, the mask implementation is blocking the data flow inside the neural network.

I encounter this problem when setting the pruning percentage to be 0.5 as shown in the readme file and I got almost 0% accuracy after first round of pruning.

Could you please advice if this is the correct method? Should I use fine tuning to recover the accuracy or I should decrease the pruning ratio first and do the pruning progressively?

As the ratio of 0.5 is suggested in the code, can I check do you also encounter the similar situation of 0% accuracy for the first round of pruning with 0.5 percentage?

Thank you so much for your reply and advice.

Eric-mingjie commented 5 years ago

This is normal. For ResNet, if all filters in a layer are pruned, you can just see this layer as an identity mapping.

zhoudaquan commented 5 years ago

This is normal. For ResNet, if all filters in a layer are pruned, you can just see this layer as an identity mapping.

What about models with no skip connection?

Eric-mingjie commented 5 years ago

Then the model is not useful.