search

microsoft / Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2
Other
1.89k stars 344 forks source link

Async allreduce for tensor-parallel #447

Open drcanchi opened 1 month ago

drcanchi commented 1 month ago

By default, Megatron-DeepSpeed disables async_tensor_model_parallel_allreduce. Is there any plan to enable this feature ?

https://github.com/microsoft/Megatron-DeepSpeed/blob/main/megatron/arguments.py#L398