Any plans for Mac/Metal support?

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs

MIT License

3.56k stars 273 forks source link

Any plans for Mac/Metal support? #184

Closed aikitoria closed 9 months ago

aikitoria commented 10 months ago

Hi, I've been using this library on a 4090 with great success (it provides absolutely best in class performance there).

I was considering getting a Mac with the M2 Ultra 128-192GB unified memory to run larger models, until I saw that this library is currently only implemented for CUDA and ROCM. Do you already have any plans to port it to Metal?

turboderp commented 9 months ago

There are currently still no plans to add Mac support, no. I don't own one and can't see myself getting one anytime soon. Feel free to take a stab at it though. :)

ehartford commented 7 months ago

what if I send you a mac? @turboderp

ehartford commented 7 months ago

it's kind of a big deal, almost all AI devs use macbooks