Mixtral MoE support?

[nivibilla]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [x] I carefully followed the README.md.
• [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [x] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Feature Description

Support for Mixtral MoE

[mathieu-heyarctic]

https://github.com/SJTU-IPADS/PowerInfer/issues/13

[BadisG]

@mathieu-heyarctic https://github.com/SJTU-IPADS/PowerInfer/issues/13 is for Mistral, not Mixtral

[dRAT3]

This would be OP