search

huggingface / candle

Minimalist ML framework for Rust
Apache License 2.0
15.12k stars 883 forks source link

Vulkan support #1810

Open RangerMauve opened 6 months ago

RangerMauve commented 6 months ago

The m2 metal support is great for people running macs, however it doesn't cover the majority of devices out there. Supporting the Vulkan graphics library can greatly increase the reach of GPUs that could be used for inference.

On a related note, llama.cpp recently landed support for it and tools like Ollama are working on integrating this support.

oddpxl commented 5 months ago

I'd like to second the wish for that - support for Vulkan in candle would be great !

niklasha commented 1 month ago

I don't want to promise too much, but I'll mention it here, just in case anybody else is planning or doing similar work: I have started to add Vulkan support to Candle in my private fork. It is not far gone, I have not even started doing any kernel work, just initializing buffers, so it's not worth to share just yet. My main motivation for this is for making Candle work on OpenBSD with Radeon, which is otherwise not supported in any way.