Research Directions

[Mistobaan]

From our website we link the following papers:

• [ ] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
  • GDrive
• [ ] Improving Transformer Models by Reordering their Sublayers
• [ ] Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• [ ] Do Transformers Need Deep Long-Range Memory?
• [ ] Reversible Layers
  • [ ] tf.mesh code
  • [ ] tf.mesh code

Tracking them here until we are at the point to experiment with them.

[StellaAthena]

You can use checkboxes instead of bullets to make list management easier:

• [X] this one has been completed
• [ ] this one has not been completed

- [X] this one has been completed
- [ ] this one has not been completed