search

evilsocket / cake

Distributed LLM and StableDiffusion inference for mobile, desktop and server.
Other
2.64k stars 141 forks source link

Thanks for the FOSS! Suggestion for future possible backends runtimes: Vulkan, OpenCL, SYCL/OpenVino/intel GPU, AMD gpu/ROCm/HIP. #20

Open ghchris2021 opened 4 months ago

ghchris2021 commented 4 months ago

Thanks for the FOSS!

Suggestion for future possible backends runtimes: Vulkan, OpenCL, SYCL/OpenVino/intel GPU, AMD gpu/ROCm/HIP.

Vulkan and OpenCL both have the possibility of being very portable to GPUs and also to some extent CPUs that have supporting SW for it.

SYCL can run on various CPU / GPU platforms; it / openvino etc. is the primary ideal target to support intel gpus.

James4Ever0 commented 4 months ago

This software is based on Candle. It is like PyTorch in Rust. So if you want more acceleration you should look for the development there.

evilsocket commented 4 months ago

i like people suggesting a bulk of new inferences backend like it's nothing :D

malikwirin commented 2 weeks ago

So GPU acceleration through ROCm has to be implemented in Candle first?

malikwirin commented 2 weeks ago

So GPU acceleration through ROCm has to be implemented in Candle first?

Their discussion about amd support can be found here

There is also a WIP CUDA implementation for non NVIDIA GPUs: https://github.com/vosen/ZLUDA