search

EricLBuehler / mistral.rs

Blazingly fast LLM inference.
MIT License
3.47k stars 248 forks source link

Feature request: Rocm support #431

Open ilovethensa opened 3 months ago

ilovethensa commented 3 months ago

Any possibility for rocm support? Or even openCL?

EricLBuehler commented 3 months ago

Hi @ilovethensa! ROCm support is definitely on the radar, however it will be in one month at earliest because of the other projects and exciting features which are upcoming. OpenCL would be helpful too, but I think we can use ZLUDA to quickly get started with ROCm support.

ilovethensa commented 3 months ago

Hi @ilovethensa! ROCm support is definitely on the radar, however it will be in one month at earliest because of the other projects and exciting features which are upcoming. OpenCL would be helpful too, but I think we can use ZLUDA to quickly get started with ROCm support.

Thank you, love the project good luck, take your time i know how much of a pain in the ass rocm can be

curvedinf commented 3 weeks ago

Hi I'm also interested in AMD support. I'd prefer OpenCL or Vulkan to ROCm since they don't require special drivers and are ballpark the same performance. Llama.cpp's vulkan implementation is about 15% slower than ROCm for me for reference.

BTW thank you for the awesome project. I'll keep my eye on it.