getumbrel / llama-gpt

A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device. New: Code Llama support!
https://apps.umbrel.com/app/llama-gpt
MIT License
10.73k stars 696 forks source link

[FEATURE REQUEST] Support for NPU hardware acceleration #157

Open Flimsy-Fox opened 5 months ago

Flimsy-Fox commented 5 months ago

With both Intel and AMD releasing CPUs with NPUs (Neural Processing Unit) in their laptop CPU designs, NPUs being a part of the Orange Pi design, and the Raspberry Pi Project launching an official NPU HAT, it's clear that the AI space is moving heavily into hardware acceleration even for low-power systems.

This feature request is for NPU hardware acceleration support.