Blog: Techniques for training large neural networks

[konabuta]

Blog: Techniques for training large neural networks

Link: https://openai.com/research/techniques-for-training-large-neural-networks

This blog explains the parallism technologies for training large neural networks.

[konabuta]

Data parallelistm

1. indendently compte the gradient on each worker.
2. average the gradents across workers.
3. independently compute the same new parameters on each worker.

Step can be bottleneck becuase it requires transferring data.

[konabuta]

Pipeline parallelism

• bubbls: a worker waits for outputs from the previous machine to be used as its inputs.

[konabuta]

Tensor parallelism

[konabuta]

Mixture-of-Experts (MoE)

[konabuta]

Others

• Checkpointing
• Mixed Precision Training
• Offloading
• Memory Efficient Optimizer
• Compression

[konabuta]

Reference

• https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/