Self-Normalizing Neural Networks

[joyhuang9473]

Self-Normalizing Neural Networks

• Author: Günter Klambauer, Thomas Unterthiner, Andreas Mayr, Sepp Hochreiter
• Origin: https://arxiv.org/abs/1706.02515
• Related:
  • https://github.com/bioinf-jku/SNNs
  • SNNs-Self-Normalizing-Neural-Networks-Caffe-Reimplementation
  • https://github.com/bioinf-jku/SNNs/blob/master/getSELUparameters.ipynb
  • https://github.com/shaohua0116/Activation-Visualization-Histogram: Compare SELUs (scaled exponential linear units) with other activation on MNIST, CIFAR10, etc.
  • https://data-sci.info/2017/06/11/%E6%9C%80%E6%96%B0%E6%BF%80%E6%B4%BB%E7%A5%9E%E7%B6%93%E5%85%83self-normalization-neural-network-selu/

The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties.

SNNs work well for architectures with many layers, allowed us to introduce a novel regularization scheme, and learn very robustly.

[joyhuang9473]

HolmesShuan/SNNs-Self-Normalizing-Neural-Networks-Caffe-Reimplementation: https://github.com/HolmesShuan/SNNs-Self-Normalizing-Neural-Networks-Caffe-Reimplementation