feat(llama.cpp): Vulkan, Kompute, SYCL

mudler / LocalAI

:robot: The free, Open Source OpenAI alternative. Self-hosted, community-driven and local-first. Drop-in replacement for OpenAI running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. It allows to generate Text, Audio, Video, Images. Also with voice cloning capabilities.

https://localai.io

MIT License

21.75k stars 1.66k forks source link

feat(llama.cpp): Vulkan, Kompute, SYCL #1647

Open mudler opened 5 months ago

mudler commented 5 months ago

Tracker for: https://github.com/ggerganov/llama.cpp/discussions/5138 and also ROCm

[x] Vulkan: https://github.com/mudler/LocalAI/pull/2648 (upstream https://github.com/ggerganov/llama.cpp/pull/2059 )
[ ] Kompute: (upstream https://github.com/ggerganov/llama.cpp/pull/4456 )
[x] sycl: https://github.com/mudler/LocalAI/pull/1660 ( upstream https://github.com/ggerganov/llama.cpp/pull/2690 )
[x] ROCm: #1595 (#1100)

RiQuY commented 3 months ago

The merge requests linked on this issue appears to be merged upstream. Does that mean LocalAI already supports Vulkan or there are any additional tasks to do before that?

mudler commented 2 weeks ago

The merge requests linked on this issue appears to be merged upstream. Does that mean LocalAI already supports Vulkan or there are any additional tasks to do before that?

Only kompute is missing as for now