DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks

shunk031 / paper-survey

📚 Survey of previous research and related works on machine learning (especially Deep Learning) in Japanese

https://shunk031.github.io/paper-survey/

151 stars 12 forks source link

Closed shunk031 closed 5 years ago

shunk031 commented 5 years ago

shunk031 commented 5 years ago

全結合からなる self-attention 層に対して dropout を行う DropAttention を用いて過学習抑制し汎化性能向上を確認した。列ベース・要素ベースでattentionをdropoutする手法を比較し、性能向上を確認。