search

FlagOpen / FlagScale

FlagScale is a large model toolkit based on open-sourced projects.
Other
132 stars 40 forks source link

[Hetero] Fix p2p comm in the hetero scenario #198

Closed heavyrain-lzy closed 2 weeks ago

heavyrain-lzy commented 3 weeks ago
  1. Fix p2p comm in the hetero scenario because the commit https://github.com/NVIDIA/Megatron-LM/commit/8af3dae72a944848db0122047d89e04ab078b178
  2. Change the parallel_context to support ulysses sequence parallel.