Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation

shunk031 / paper-survey

📚 Survey of previous research and related works on machine learning (especially Deep Learning) in Japanese

https://shunk031.github.io/paper-survey/

151 stars 12 forks source link

Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation #129

Closed shunk031 closed 7 years ago

shunk031 commented 7 years ago

https://arxiv.org/abs/1708.03769v1

shunk031 commented 7 years ago

学習中にsoftmax後の活性化状態がサチると勾配の伝達を妨げ、悪い局所解に落ちる要因となる。そのため、fc -> add noise -> softmax とすることで、サチりずらくすると良い局所解に収束し汎化性能もあがる。 https://twitter.com/asam9891/status/900250424158044160

shunk031 commented 7 years ago

学習時にsoftmaxへの入力に対して効果的にノイズを加えることで活性化状態の飽和を防ぎ，モデルの汎化性能を上げるNoisy Softmaxを提案．

paper summary