I think the formula which is noted is correct - Githubissues

jiecaoyu / XNOR-Net-PyTorch

PyTorch Implementation of XNOR-Net

472 stars 119 forks source link

I think the formula which is noted is correct #117

Open lixcli opened 3 years ago

lixcli commented 3 years ago

Hi, I think the original formula which is noted is correct. Here is my demonstrate.

Firstly, the primary formula is

$\frac{\partial C}{\partial W_i} = \frac{\partial C}{\partial \widetilde{W_i}} \times (\alpha \times \frac{sign(W_i)}{\partial W_i} + \frac{1}{n})$

In your formula, because each Wi in |Wi| is independent ,the formula could be transformed into

$\frac{\partial C}{\partial W_i} = \frac{\partial C}{\partial \widetilde{W_i}} \times (\alpha \times \frac{sign(W_i}{\partial W_i} + \frac{1}{n} \times sign(W_i) \times \frac{|W_i|}{W_i})$

there is a absolute value in it because

$\alpha = \frac{\sum |W_i|}{n}$

and we can get another formula by a simple demonstrate.

$\frac{1}{n} \times sign(W_i) \times \frac{|W_i|}{W_i}= \frac{1}{n}$

So, the primary formula is correct

$\frac{\partial C}{\partial W_i} = \frac{\partial C}{\partial \widetilde{W_i}} \times (\alpha \times \frac{sign(W_i)}{\partial W_i} + \frac{1}{n})$