Open Meteor-Stars opened 3 years ago
Hi, Meteor,
I am very glad that you are interested in this paper. Your questions are very representative!
For the first question, yes. There are two ways of pruning, either pruning neurons or weights, where the first choice leads to more structural accelerations and the second one accepts a larger sparsity. setting weights to 0 alone cannot make the model smaller and we need to reformulate the model based on the structure based on a sparse model. For example, for a sparse matrix, it still has the same size as the dense one, but if you changed it to compressed row storage, then the size is greatly reduced.
Second. We prune the weights during the pruning, it will be active (non-zero) again, so in the final iteration, we also need to prune it to make sure sparsity.
Third. You didn't ask this question, but I will answer it for extending this work. Pruning based on probability is more appealing and wasn't achieved during the submission of the NeurIPS'19 paper. You can achieve it by forgetting about the inverse gamma prior and beta prior for the updates of \sigma^2 and \delta in Eq.(14-15) in the paper.
Hope these answers are helpful.
Hi, Meteor,
I am very glad that you are interested in this paper. Your questions are very representative!
For the first question, yes. There are two ways of pruning, either pruning neurons or weights, where the first choice leads to more structural accelerations and the second one accepts a larger sparsity. setting weights to 0 alone cannot make the model smaller and we need to reformulate the model based on the structure based on a sparse model. For example, for a sparse matrix, it still has the same size as the dense one, but if you changed it to compressed row storage, then the size is greatly reduced.
Second. We prune the weights during the pruning, it will be active (non-zero) again, so in the final iteration, we also need to prune it to make sure sparsity.
Third. You didn't ask this question, but I will answer it for extending this work. Pruning based on probability is more appealing and wasn't achieved during the submission of the NeurIPS'19 paper. You can achieve it by forgetting about the inverse gamma prior and beta prior for the updates of \sigma^2 and \delta in Eq.(14-15) in the paper.
Hope these answers are helpful.
Hi, thank you very much for your reply, which solved my confusion and gave me great inspiration. The work of this paper is great. If there are other questions about the paper, I will come back. Thank you very much again! Best wishes!