hardware: Intel iGPU, dGPU and NPU support

[xiangyang-95]

Overview

• Intel's Lunar Lake is releasing soon, which has CPU, NPU and iGPU in a single chip

Tasklist

• [x] https://github.com/janhq/cortex.cpp/issues/677
• [x] https://github.com/janhq/cortex.llamacpp/issues/107
• [ ] https://github.com/janhq/cortex.llamacpp/pull/99 from @xiangyang-95

Original Post

Problem Unable to use Intel integrated GPU and discrete GPU to offload the model inferencing.

Success Criteria Able to use Intel integrated GPU and discrete GPU to offload the model inferencing.

Additional context I can add in the documentation to add Intel GPU support to nitro inference server.

[rahulunair]

I am trying to do the same, this would be incredibly useful with the latest Intel Core Ultra chips as well, as there is a unified cpu + GPU architecture.

With this, any user using jan or nitro directly on latest intel core ultra cpus will get a boost from the gpu

Options:

1. Intel acceleration support for llama.cpp using ipex-llm as a backend: https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html

2. Direct compilation of llama.cpp with sycl bindings on device that support sycl: https://github.com/ggerganov/llama.cpp/blob/master/README-sycl.md

[xiangyang-95]

@rahulunair I have integrated the nitro server with Intel OneAPI SYCL Optimization on llama.cpp. Stay tuned for the pull request.

[0xSage]

closing as dupe of #677

[dan-homebrew]

I'm re-opening this, given our discussions with Intel. We should evaluate the following possibilities:

• llama.cpp through sycl (preferred)

• 677 (which Intel recommended on their call)

• IPEX-LLM (less preferred to OpenVino)