search

Eric-mingjie / rethinking-network-pruning

Rethinking the Value of Network Pruning (Pytorch) (ICLR 2019)
MIT License
1.51k stars 293 forks source link

Network slimming for ANN on MNIST #19

Open ArvindSubramaniam opened 5 years ago

ArvindSubramaniam commented 5 years ago

How to do the pruning step for batchNorm1D layers in an ANN, where you would be using the weights directly rather than using a mask? If possible, a sample code on the nn.Linear layer and the BatchNorm1d layer would be really helpful!

When I use the same code for batchNorm1D, I get :

Traceback (most recent call last): File "MLPprune.py", line 156, in end_mask = cfg_mask[layer_id_in_cfg] IndexError: list index out of range

Eric-mingjie commented 5 years ago

Which pruning method are you using?

ArvindSubramaniam commented 5 years ago

Network slimming. Sorry I forgot to mention that. (Given that I am using an ANN with linear layers and batchNorm1d layers)

Eric-mingjie commented 5 years ago

For BatchNorm1d, you can refer to Network Slimming code for ImageNet.

ArvindSubramaniam commented 5 years ago

Thanks...but I wanted to know if it would be possible to write a code for the BatchNorm1d and the nn.Linear layers in the MLP without using a mask.

arvind1096 commented 5 years ago

Also, in main_finetune.py, under args.refine, you have loaded the pruned model to finetune it and written : model = models. dict args.arch

What doesn't this statement lead to a size mismatch? The model would have the same number of channels as vgg, but the vggprune you are loading has lesser number of channels in each layer right?

Eric-mingjie commented 5 years ago

@arvind1096 Can you point to which pruning methods you are using, or which files you are running?

arvind1096 commented 5 years ago

Thanks for asking. I resolved the issue just 2 days ago. However, while I was trying to come up with MLPprune.py using the code in "Rethinking-network-pruning/cifar/network-slimming/vggprune.py ",

after applying threshold 'thre' as mentioned in vggprune.py, my network is removing almost all of the connections. For instance, if I had 200K connctions before pruning, I am only getting 36 connections after pruning the MLP.

Note: I have not pruned the MLP, but merely want to preserve the nodes whose BatchNorm scaling factors do not cross the threshold, so that the dimension of each layer does not change. Hence, I have a binary network of 1s and 0s, where 1s are the preserved connections and everything else is zero .(only 36 ones remain in my case)

I am doing a project which takes in the MLp as a graph and forms an adjacency matrix from it. I dont remove the zeroed out nodes, since that would change the total number of nodes => change the shape of the adjacency matrix. If you could share/upload the code for the pruned MLP model, it would really help me progress!

If possible could you attach a code for getting MLPprune.py for "cifar/network-slimming/" so that I can get the new MLP which has almost half the number of 1s as before? (assuming args.percent is 0.5)

Eric-mingjie commented 5 years ago

Can you give me the code for your MLP network or describe it in detail? I am still not sure about the MLP structure you mentioned.

arvind1096 commented 5 years ago

MLP_adjacency Matrix.docx