CPT: add more args and exec scripts

pjlab-sys4nlp / llama-moe

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

https://arxiv.org/abs/2406.16554

Apache License 2.0

849 stars 44 forks source link

Closed Spico197 closed 9 months ago

Spico197 commented 9 months ago

add final data portion of sheared llama
add gate_network_type, moe_calculator_score_scale_factor, and update prob_map arguments in config
add exec scripts

DaizeDong commented 9 months ago

well done