Open cwh1981 opened 3 years ago
attention
self attention
encoder / decoder
label smoothing : label noisy case
Attention is all you need : https://arxiv.org/pdf/1706.03762.pdf
http://jalammar.github.io/illustrated-transformer
https://medium.com/@adityathiruvengadam/transformer-architecture-attention-is-all-you-need-aeccd9f50d09
Neural Machine Translation by jointly learninig to align and translate ( https://arxiv.org/pdf/1409.0473.pdf )
Vision Transformer
position embedding
2015), ReLU (Nair & Hinton, 2010), local contrast normalization (Jarrett et al., 2009) and max-pooling (Scherer et al., 2010) layers