Open wjsteve opened 3 years ago
Hello. I got a question while reproducing your interesting experiment.
In section 2 of https://arxiv.org/pdf/1708.06519.pdf, "Scaling Factors and Sparsity-induced Penalty" shows below equation.
Question: g(γ) means L1 norm, but https://github.com/Eric-mingjie/rethinking-network-pruning/blob/master/imagenet/network-slimming/main_finetune.py#L187 applys torch.sign like "m.weight.grad.data.add_(sparsity * torch.sign(m.weight.data))" not L1 norm
So isn't it right to use "m.weight.grad.data.add_(sparsity * m.weight.data.abs()))" for updateBN?
The gradient of l1 norm is the sign function.
Hello. I got a question while reproducing your interesting experiment.
In section 2 of https://arxiv.org/pdf/1708.06519.pdf, "Scaling Factors and Sparsity-induced Penalty" shows below equation.![image](https://user-images.githubusercontent.com/11556141/109136837-e45f0200-779b-11eb-9f04-de82ddd6d92f.png)
Question: g(γ) means L1 norm, but https://github.com/Eric-mingjie/rethinking-network-pruning/blob/master/imagenet/network-slimming/main_finetune.py#L187 applys torch.sign like "m.weight.grad.data.add_(sparsity * torch.sign(m.weight.data))" not L1 norm
So isn't it right to use "m.weight.grad.data.add_(sparsity * m.weight.data.abs()))" for updateBN?