BaguaSys / bagua

Bagua Speeds up PyTorch
https://tutorials-8ro.pages.dev/
MIT License
872 stars 83 forks source link

support gradient accumulation #369

Closed wangraying closed 2 years ago

wangraying commented 2 years ago

refer to: https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html#DistributedDataParallel.no_sync

shjwudp commented 2 years ago

it is now supported by https://github.com/BaguaSys/bagua/pull/312