Where is the TurboSparse-Mixtral mlp_predictor?

SJTU-IPADS / PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

MIT License

7.98k stars 415 forks source link

Where is the TurboSparse-Mixtral mlp_predictor? #203

Open MatthewCroughan opened 5 months ago

MatthewCroughan commented 5 months ago

Predictors for all other models are provided, but not this one https://huggingface.co/PowerInfer?search_models=predictor

I was wanting to convert TurboSparse-Mixtral into a quantized GGUF to reproduce the claims made about usage on smartphones.

yichen0104 commented 5 months ago

Similar situation here. Since it is basically Bamboo with an extra MLP as predictor within BambooMLP, I suspect we need standalone pretrained predictor weights s.t. they can be combined with the TurboSparse model.