[Section] Demo introduction

[justheuristic]

https://github.com/training-transformers-together/hf-website-how-to-join

Demo page (updated on push): https://training-transformers-together.github.io/

• [x] intro and motivation text

• [x] live training plot

• [x] model & dataset info

• final polishing:

  • [x] add company/university logos to the demo header
  • [ ] update footer links (e.g. twitter & paper)
  • [ ] remove the full demo will be posted header once the content is there
  • [ ] link final plot with main training run (#6 )
  • [ ] try adding bubble chart or leaderboard-list using a background thread, make sure performance is alright
  • [ ] review round on monday (ask around the slack channel)
  • [ ] offsets, margins

[justheuristic]

@TimDettmers PLZ review up until "how do i join"

[TimDettmers]

I think the intro would be better if it goes straight to the point in the first sentence. I think also a bit more formal/neutral style could be better for the first few sentences in the demo. I would also like it to be a bit more like an abstract. Problem + importance + solution (+description) + results. Something like:

"For individual researchers and small labs, pretraining state-of-the-art vision and language model is no longer feasible due to the massive computational resources required for pretraining. For example, pretraining GPT-3 on a single GPU server would take decades. Training Transformers Together is an initiative to perform pretraining of large models on commodity hardware coordinated through the Hugging Face community. We use the decentralized training framework, Hivemind, to perform efficient training over the internet. In our demo, we demonstrate our approach by pretraining [your text copied after this] DALLE: a transformer "language model" that generates images from text description. We're training this model on LAION - the world's largest openly available image-text-pair dataset with 400 million samples. Our model is based on dalle-pytorch with several tweaks for memory-efficient training."