Bug/fix hybrid tp dp - Githubissues

xrsrke / pipegoose

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

MIT License

76 stars 17 forks source link

Open 3outeille opened 9 months ago

3outeille commented 9 months ago

EDIT:
- Found out fairscale has the same bug as us => https://github.com/xrsrke/megatron-tp . Issue is put on hold
Main contributions:
- [ ] TP convergence work (cf wandb log)
- [ ] DP + TP + Zero1 convergence work (cf [wandb log]())

review-notebook-app[bot] commented 9 months ago

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.