SJTU-IPADS / PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

MIT License

7.96k stars 412 forks source link

Windows CPU/GPU support #86

Closed hodlen closed 10 months ago

hodlen commented 10 months ago

It requires modifying a series of POSIX API calls that are supported by MSVC to build PowerInfer on Windows under CPU inference and hybrid inference mode, including:

Atomic operations in stdatomic.h;
File system APIs such as dirname;
system call also requires testing to ensure Python is running in the correct path and inherits all environment variables.

Related PR: ~~#73 for CPU support~~ (closed)

114 for CPU/GPU support

18635191739 commented 10 months ago

can windows use powerinfer now?

hodlen commented 10 months ago

Experimentally supported Windows on #114 now. We will spend some time to optimize inference performance under hybrid (CPU + GPU) inference and then merge it.