[FEATURE]: Gradient accumulation with Gemini DDP

hpcaitech / ColossalAI

Making large AI models cheaper, faster and more accessible

https://www.colossalai.org

Apache License 2.0

38.7k stars 4.34k forks source link

[FEATURE]: Gradient accumulation with Gemini DDP #3375

Open zixiliuUSC opened 1 year ago

zixiliuUSC commented 1 year ago

Describe the feature

currently according to Gemini's official desc, we cannot do gradient accumulation manually, hope Colossal AI team can add this feature to the projest.

binmakeswell commented 1 year ago

Hi @zixiliuUSC Received, will consider in follow-up. Thanks for the feedback.