search

pjlab-sys4nlp / llama-moe

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)
https://arxiv.org/abs/2406.16554
Apache License 2.0
879 stars 46 forks source link

Moefication: Expert Load visualization & MoE gate evaluation bug fix #21

Closed DaizeDong closed 1 year ago