Feature Request: NPU Support

[gformcreation]

Prerequisites

• [X] I am running the latest code. Mention the version if possible as well.
• [X] I carefully followed the README.md.
• [X] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [X] I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

First, thank you for your incredible work on this project! To enhance its performance, especially on mobile devices and NPU-enabled PCs like those with Copilot+, I would love to see support for Neural Processing Units (NPUs).

Motivation

Integrating NPU support would significantly improve the speed and efficiency of AI tasks, offering users a smoother and more responsive experience in terms of quick response generation.

Possible Implementation

No response

[piDack]

AMD’s NPU has an implementation in this repository, but its performance is poor. I’ve done some exploration, but I couldn’t even pass the unit tests for basic op, so I believe that support for AMD’s NPU might take a long time, unless AMD deems it worth the effort.

[hgftrdw45ud67is8o89]

volunteer to test intel AI boost (NPU) here.

[hpvd]

AMD's NPU is XDNA, for current state see https://github.com/ggerganov/llama.cpp/issues/1499

[hpvd]

For INTEL's NPU this may be the right issue https://github.com/ggerganov/llama.cpp/issues/5079

[hpvd]

QUALCOMM's NPU is Hexagon, see https://github.com/ggerganov/llama.cpp/issues/2687

[FranzKafkaYu]

can we support more NPU liked devices like MediaTek APU(AI Processing Unit),its development kit named NeuroPilot SDK,link

[themoneyevo]

any update？