Can Powerinfer run on CPUONLY?

SJTU-IPADS / PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

MIT License

7.96k stars 412 forks source link

Closed 0wwafa closed 4 months ago

0wwafa commented 4 months ago

How does Powerinfer compare to llama.cpp when run on CPU only?

YixinSong-e commented 4 months ago

PowerInfer can work well on x86 CPU only mode. For other like arm, PowerInfer has limited speedup for now. :)