search

mil-ad / snip

Pytorch implementation of the paper "SNIP: Single-shot Network Pruning based on Connection Sensitivity" by Lee et al.
MIT License
103 stars 15 forks source link

dose weights real pruned? #5

Closed Pong-97 closed 3 years ago

Pong-97 commented 3 years ago

Hi! weights and its gradients are set to 0 in the code which i think is different from real pruning.

mil-ad commented 3 years ago

What do you mean by real pruning? Are you referring to "structured" pruning?

Snow-Crash commented 3 years ago

I guess what @Pong-97 mean by 'real pruning' is that when a connection is pruned, it should not appear in weight matrix or the weight matrix should be smaller, but '0' means the connection is still there, it's not actually pruned.

To avoid confusion, imagine the dnn as a graph, if there is an edge from neuron i to j, and its weight is 0, we call this connection 'logically' does not exist. If there is no such edge from neuron i to j at all, i.e. no underlying data structure or memory to store this connection, there is no physical connection from i to j.

The point is, deep learning frameworks mostly operate on dense matrix format, for irregular pruning, there is no good way to construct a model in which those pruned connection does not 'physically' exist. But logically, multipliying neuron activation with 0 is equivalent as the connection does not exist.

If you mean real pruning by 'after pruning the connection does not even physically exist', I don't think any off-the-shelf dl framework can support this well. Indeed, one can develop a inference model using sparse matrix library to achieve this.

Structured pruing is a special case, suppose a conv layer kernel is 3x3x8x16, after pruning, 4 filters are removed. you can actually build a new conv layer with kernel 3x3x8x12, then use the pruned old kernel to initialize the smaller new kernel by skipping 0 filters in pruned kernel.

Pong-97 commented 3 years ago

What do you mean by real pruning? Are you referring to "structured" pruning?

Thank you for your reply!

As @Snow-Crash say, i think i confused the concept of "structured" and "unstructured" pruning.

thanks!

Pong-97 commented 3 years ago

@Snow-Crash thank you for your reply!your answer perfectly solved my doubts!

now i get better understand about it!

respct!