Curated NLP Primers

[manisnesan]

📚 Curated NLP Primers

➡️ Primers for common NLP topics: Attention, Autoregressive vs. Autoencoder Models, Token Sampling Methods, Transformers, BERT

🔹 Attention: http://attention.aman.ai/

• Classic Sequence-to-Sequence Model
• Sequence-to-Sequence Model with Attention
• Context Vector
• Compared: Attention Mechanism and Fixed-length Context Vector Approach

🔹 Autoregressive vs. Autoencoder Models: http://enc-dec.aman.ai/

• How do Autoregressive/Decoder Models Work?
• How do Autoencoder/Encoder Models Work?
• Encoder-Decoder/Seq2Seq Models
• Case Study: XLNet - the Best of Both Worlds (Permutation Language Modeling)

🔹 Token Sampling Methods: http://token-smple.aman.ai/

• Background: Logits and Softmax
• Related: Temperature
• Greedy Decoding
• Exhaustive Search Decoding
• Beam Search
• Constrained Beam Search
• Top-k Sampling
• Top-p Sampling (Nucleus Sampling)

🔹 Transformers: http://transformer.aman.ai/

• Mathematical Background (Vectors, Matrix Multiplication, Dot Product, Masking, Sampling)
• Attention (Additive/Multiplicative/Dot Product Attention, Self/Cross-Attention, Multihead Attention)
• Core Components of the Transformer Architecture (Embeddings, Positional Encoding, Skip Connections, Layer Normalization, Softmax)
• Top-level Transformer Architecture (Encoder and Decoder stack)
• Implementation details (Byte-Pair Encoding, Teacher Forcing, Label Smoothing)
• Lessons Learned (What are Transformers learning? Why is training them so hard?)
• Pros/cons of Transformers relative to CNNs/RNNs
• Relation between Transformers and Graph Neural Networks

🔹 BERT: http://bert.aman.ai/

• Background: Pre-Training
• Contextualized Embeddings
• Masked Language Modeling (MLM)
• Next Sentence Prediction (NSP)
• BERT’s Encoder Architecture vs. Other Decoder Architectures
• The Strength of Bidirectionality
• Supervised Fine-Tuning

Source: LinkedIn