search

evilsocket / cake

Distributed LLM and StableDiffusion inference for mobile, desktop and server.
Other
2.44k stars 127 forks source link

Thanks for the FOSS! Suggestion for future possible backends runtimes: Vulkan, OpenCL, SYCL/OpenVino/intel GPU, AMD gpu/ROCm/HIP. #20

Open ghchris2021 opened 1 month ago

ghchris2021 commented 1 month ago

Thanks for the FOSS!

Suggestion for future possible backends runtimes: Vulkan, OpenCL, SYCL/OpenVino/intel GPU, AMD gpu/ROCm/HIP.

Vulkan and OpenCL both have the possibility of being very portable to GPUs and also to some extent CPUs that have supporting SW for it.

SYCL can run on various CPU / GPU platforms; it / openvino etc. is the primary ideal target to support intel gpus.

James4Ever0 commented 1 month ago

This software is based on Candle. It is like PyTorch in Rust. So if you want more acceleration you should look for the development there.

evilsocket commented 1 month ago

i like people suggesting a bulk of new inferences backend like it's nothing :D