search

SafeAILab / EAGLE

Official Implementation of EAGLE-1 (ICML'24) and EAGLE-2 (EMNLP'24)
https://arxiv.org/pdf/2406.16858
Apache License 2.0
780 stars 79 forks source link

Release Mixtral Training Code #31

Closed tjtanaa closed 7 months ago

tjtanaa commented 8 months ago

Hi, I am interested in training Mixtral Eagle. Could I know will there be plans to release the training code anytime soon?

beginOfAll commented 8 months ago

me too. 我准备使用中文sft数据训练一下Mixtral Eagle，并测试下在中文任务下的加速效果。

hongyanz commented 8 months ago

We will upload the training code for Mixtral in a few days. Please stay tuned. Thanks for your interest.

Liyuhui-12 commented 7 months ago

The current weights were trained using Mixtral_8x7B.json. We have not optimized for MoE yet, so the same training code (main.py) was used. We are planning to optimize for MoE.