optimizer offloading 太强了

alibaba / Pai-Megatron-Patch

The official repo of Pai-Megatron-Patch for LLM & VLM large scale training developed by Alibaba Cloud.

Apache License 2.0

714 stars 101 forks source link

Open 154912369 opened 3 months ago

154912369 commented 3 months ago

可以相对低资源的训练较大模型了，感谢大佬们

jerryli1981 commented 3 months ago

谢谢鼓励和支持哈

WuNein commented 1 week ago

H100 开OPTIMIZER_OFFLOAD = auto 性能损失非常小！尤其是batch size比较大的时候！