issues
search
xrsrke
/
pipegoose
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
MIT License
76
stars
17
forks
source link
Reproducible in 3D Parallelism
#15
Closed
xrsrke
closed
9 months ago
xrsrke
commented
10 months ago
Reading
https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/tensor_parallel/random.py
Reading