anyone try theCenteredWN on LSTM?

huangleiBuaa / CenteredWN

This project is the Torch implementation of our ICCV 2017 paper: Centered Weight Normalization in Accelerating Training of Deep Neural Networks

BSD 2-Clause "Simplified" License

21 stars 5 forks source link

anyone try theCenteredWN on LSTM? #3

Open hefeiwangyande opened 5 years ago

hefeiwangyande commented 5 years ago

Hi,

The author mentioned in the paper that this CWN operation can be used as a basic linear module. Does anyone experiment on lstm?

Thank you!

huangleiBuaa commented 5 years ago

HI, @hefeiwangyande Sorry for the late reply, since I didnot manage my github for a long time. I did try on the RNN for char-RNN classification in 2017. CWN has no significant advantage than Weight Normalization(WN), when used on all layers. I conjectured the reason is that the weight on RNN is shared and the Network is not that big/deep, since CWN provides more constraints on weights (e.g, centering) compared to WN. I guess if one runs a very large LSTM/RNN, CWN may be better than WN.