issues
search
xrsrke
/
pipegoose
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
MIT License
76
stars
17
forks
source link
Making pipeline parallelism compatible with `transformers`
#52
Open
xrsrke
opened
9 months ago
xrsrke
commented
9 months ago
TODO
[ ] Use pp size = 2 in test
[ ] Add running test in NCCL backend
Debug Notes
[ ] Run autograd on the target cuda device and get results from that
TODO
Debug Notes