Transformer Day Chapter

[ramon-astudillo]

Create the labs guide for the Transformers day under pages/transformers/. This should include

• [ ] Introduction and explanation of the Transformer architecture (both encoder and decoder)

  • [x] Add a draft of Transformer formulation (feel free to modify) @ramon-astudillo (expected today)
  • Check other days as reference

• [ ] Explanation in detail of the attention mechanism. Maybe include some plots of attention, in particular causal attention. Attention is the most important part of Transformers so it's worth expanding this in detail. Maybe positional embeddings also deserve some love.

The day will be centered in the decoder and we can leave the encoder-decider as a final section with no exercises for this year.

• [ ] Exercise code blocks copying the code from https://github.com/LxMLS/lxmls-toolkit/issues/178 (see other days as example)

  • [ ] put placeholders for exercise blocks (for now all in the decoder section).

• [ ] Explanations (Context for the exercises)

• [ ] More advanced information: Open to suggestions @tmynn @lhaausing . If we complete fine-tuning in https://github.com/LxMLS/lxmls-toolkit/issues/177. We could add some explanations about this (instruction tuning?)

branch: https://github.com/LxMLS/lxmls-guide/tree/transformer-day

Expected Finishing date:

• June 12 - June 19 for the Intro and explanation of architecture
• We depend on the other issues finishing for the rest, but we can have placeholders for exercises and complete as we go. I'd say before June 10

[ramon-astudillo]

Added first draft, see https://github.com/LxMLS/lxmls-guide/blob/transformer-day/pages/transformers.tex