VainF / Torch-Pruning

[CVPR 2023] DepGraph: Towards Any Structural Pruning
https://arxiv.org/abs/2301.12900
MIT License
2.73k stars 337 forks source link

What exactly does this method prune? #332

Open AFallDay opened 10 months ago

AFallDay commented 10 months ago

After reading the paper it never made sense that structured pruning was pruning what, parameters, weights? I think it's a bit abstract, please answer my questions, thanks!

janthmueller commented 10 months ago

Weights are pruned in a structured manner that is considered "physical." In this context, "physical" pruning refers to the actual removal of tensor entries rather than the common approach of masking with zeros or keeping track of indices, as often seen in unstructured pruning methods.

To illustrate, imagine a weight tensor for a convolutional layer with specifications (output channels=265, input channels=64, kernel size=3x3). When channels are pruned, it means removing them from the tensor (128, 64, 3, 3). This direct elimination of weights contributes to a more straightforward and resource-efficient model compared to methods involving masked values or preserved indices.