Add dropout to Transformer implementation

[iislucas]

Transformer implementation: https://github.com/PAIR-code/tiny-transformers/blob/main/animated-transformer/src/lib/transformer/transformer_gtensor.ts

Idea: provide a boolean configuration variable to enable/disable dropout. e.g. like positional encodings: https://github.com/PAIR-code/tiny-transformers/blob/main/animated-transformer/src/lib/transformer/transformer_gtensor.ts#LL59C8-L59C27

TODOs:

• [ ] identify what drop should be applied to in the transformer. (starting point: T5 implementation: https://arxiv.org/abs/1910.10683 and earlier attention is all you need paper: https://arxiv.org/pdf/1706.03762.pdf )
• [ ] Add flags to control for dropout, set default values.
• [ ] Make the flags turn on/off dropout.
• [ ] Add a small test
• [ ] See what happens when you apply it on models of a few different sizes.

Context:

• An intro to Dropout: https://towardsdatascience.com/dropout-in-neural-networks-47a162d621d9

[1wheel]

fwiw here's the haiku implementation that Asma and I have been working off of.

We've mostly been turning it off — I think it leads to redundancy/backup circuits.