Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
76
stars
17
forks
source link
DiLoCo replication (DiLoCo: Distributed Low-Communication Training of Language Models) #59
Open
xrsrke opened 9 months ago