[Not Issue] Will you try MOE like Mixtral-8X7B?

FudanDISC / DISC-FinLLM

DISC-FinLLM，中文金融大语言模型（LLM），旨在为用户提供金融场景下专业、智能、全面的金融咨询服务。DISC-FinLLM, a Chinese financial large language model (LLM) designed to provide users with professional, intelligent, and comprehensive financial consulting services in financial scenarios.

https://fin.fudan-disc.com/

Apache License 2.0

564 stars 68 forks source link

[Not Issue] Will you try MOE like Mixtral-8X7B? #25

Open ALLinLLM opened 6 months ago

ALLinLLM commented 6 months ago

Since there are 4 experts adaptors using lora SFT in the paper, the next question is why not try MOE like Mixtral-8X7B?

598042621 commented 3 months ago