SJTU-IPADS / PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs
MIT License
7.96k stars 412 forks source link

About powerinfer-2 #207

Open Ther-nullptr opened 4 months ago

Ther-nullptr commented 4 months ago

Prerequisites

Before submitting your issue, please ensure the following:

Feature Description

I have read the paper PowerInfer-2: Fast Large Language Model Inference on a Smartphone. Will the related code open-sourced?

By the way, the core innovation of the work is how to use the heterogeneous compute system on mobile phone to run a large model. Are there any tutorials about how to use NPU/GPU of Snapdragon 8 Gen 3?