Video: AI Forum 2023 | Future of Foundation Models

[konabuta]

AI Forum 2023 | Future of Foundation Models

Link: https://www.youtube.com/watch?v=f6m0MpbNicU&list=WL&index=15&t=9s&ab_channel=MicrosoftResearch

This presentation starts by raising questions about the cost of LLMs compared to human brain.

[konabuta]

Sparsity

• MoE
  • https://arxiv.org/abs/1701.06538
  • https://openai.com/research/techniques-for-training-large-neural-networks
• Sparse Attentions
  • https://arxiv.org/abs/1904.10509
  • https://arxiv.org/abs/2307.02486

[konabuta]

RetNet

• Future base architecture of Foundation Models will be 1-bit RetNet (RetNet + BitNet)

Inference cost, Parallelism, Performance ...

BitRet

Low-bit training is not stable. And quantization aftter training shows poor performance significantly.

Overview of BitNet

Architecture details