Add MoEs support - Githubissues

huggingface / nanotron

Minimalistic large language model 3D-parallelism training

Apache License 2.0

1.14k stars 107 forks source link

Closed NouamaneTazi closed 7 months ago

NouamaneTazi commented 7 months ago

Adds support for MoEs inspired by transformers and megablocks implems