Where is the TurboSparse-Mixtral mlp_predictor?

SJTU-IPADS / PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

MIT License

7.9k stars 406 forks source link

Where is the TurboSparse-Mixtral mlp_predictor? #203

Open MatthewCroughan opened 3 months ago

MatthewCroughan commented 3 months ago

Predictors for all other models are provided, but not this one https://huggingface.co/PowerInfer?search_models=predictor

I was wanting to convert TurboSparse-Mixtral into a quantized GGUF to reproduce the claims made about usage on smartphones.

yichen0104 commented 3 months ago

Similar situation here. Since it is basically Bamboo with an extra MLP as predictor within BambooMLP, I suspect we need standalone pretrained predictor weights s.t. they can be combined with the TurboSparse model.