search

KoboldAI / KoboldAI-Client

For GGUF support, see KoboldCPP: https://github.com/LostRuins/koboldcpp
https://koboldai.com
GNU Affero General Public License v3.0
3.51k stars 758 forks source link

feature: adding 'mps' backend for GPU acceleration on macOS m1 #268

Open bozo32 opened 1 year ago

bozo32 commented 1 year ago

aiserver.py is built around cuda. This was totally reasonable till about 3 months ago.

Now there is PyTorch support for m1 hardware This requires recognition of torch.device('mps')...which does not have any equivalent to torch.cuda.device_count() nor does it seem to have anything like torch.cuda.get_device_name so a simple substitution is not possible.

There are instructions here: https://sebastianraschka.com/blog/2022/pytorch-m1-gpu.html and more detail here https://github.com/pytorch/pytorch/wiki/MPS-Backend

I have fixed huggingface.yml so that it produces a working installation (removing the CUDA dependency and adding a bunch of tensor stuff). It is, predictably, embarrassingly slow as it does not detect a GPU and defaults to CPU for everything.

I am not good enough to fix aiserver.py so that it supports the mps backend

bozo32 commented 1 year ago

as mentioned in reddit, if the code was made a bit more idiot friendly, I'd be happy to chew on making something that worked on my m1.

henk717 commented 1 year ago

We are currently working on a model backend overhaul to make it more workable. I can let you know once that is finished.

bozo32 commented 1 year ago

Please do. If there is some way to do the coding in a manner that is a bit more newbie friendly and gpu agnostic that would help me...and likely other folks when/if Nvidia starts to have competitors (if ever).

henk717 commented 1 year ago

GPU agnostic we can't easily do since the entire ecosystem was made for CUDA and AMD was wise enough to adapt their version to be compatible with the cuda code. It would have to be added seperately and maintained by someone who has the hardware.

bozo32 commented 1 year ago

yah...apple not known for humble. More like the opposite salted with extra blindingly arrogant. I'll see what I can do once you are done.

SkylerCorgi commented 1 year ago

Please let me know if I can help at all. I have an MacBook Pro with M2 Pro and would be happy to assist where I can.