search

pjlab-sys4nlp / llama-moe

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
https://arxiv.org/abs/2406.16554
Apache License 2.0
838 stars 44 forks source link

#Feature Request# Accelerated Deployment. #51

Open Xingxiangrui opened 8 months ago

Xingxiangrui commented 8 months ago

Accelerated Deployment.. Regarding the current llama, there are many mature acceleration frameworks, such as LMDeploy, vLLM, and AWQ. After using the MoE architecture, how to use these acceleration frameworks for deployment? Has adaptation been done?

Spico197 commented 8 months ago

Hi there, thanks for the question~

It is possible to use these frameworks for LLaMA-MoE inference acceleration, and we are working on it. It may take some time for development and testing. Please hang on tight ❤️

Please don't close this issue until we have further progresses. Thank you for your patience~

Xingxiangrui commented 8 months ago

If I understand correctly, the model structure of llama-MoE is identical to the the model structure of Mixtral-MoE ( negelecting the sliding window attention). You can reuse the model structure of Mixtral-MoE for adaptation. Mixtral-MoE has already been deployed with vLLM and AWQ.