search

xrsrke / pipegoose

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
MIT License
76 stars 17 forks source link

[BUG] Fix the bug where tokens can't be dispatched when the input has… #51

Closed xrsrke closed 9 months ago

xrsrke commented 9 months ago

… a batch size of more than 1