what if weight normalization combining with batch normalization?

openai / weightnorm

Example code for Weight Normalization, from "Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks"

https://arxiv.org/abs/1602.07868

MIT License

365 stars 118 forks source link

what if weight normalization combining with batch normalization? #5

Closed ZhimingZhou closed 7 years ago

ZhimingZhou commented 7 years ago

Hi, it seems we can do batch normalization, and the at the same time, re-parameterize the weight, for better/robust optimization. Would batch normalization help, given we did the re-parameterization?

Do you have some experiments on this?