Open QPHutu opened 3 weeks ago
Describe the bug
In megatron/core/pipeline_parallel/schedules.py, finish_embedding_wgrad_compute should appear before enable_grad_sync and grad_sync_func?
megatron/core/pipeline_parallel/schedules.py
finish_embedding_wgrad_compute
enable_grad_sync
grad_sync_func
Expected behavior Gradient all-reduce should happen after gradient computations.
@sanandaraj5597 Can you share some comments? Thank you!
Describe the bug
In
megatron/core/pipeline_parallel/schedules.py
,finish_embedding_wgrad_compute
should appear beforeenable_grad_sync
andgrad_sync_func
?Expected behavior Gradient all-reduce should happen after gradient computations.