search

SJTU-IPADS / PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs
MIT License
7.9k stars 406 forks source link

Support converting TurboSparse mistral model into PowerInfer GGUF #212

Open hodlen opened 2 months ago

hodlen commented 2 months ago

Support converting TurboSparse mistral model which embeds MLP in Pytorch tensors.