search

unslothai / unsloth

Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory
https://unsloth.ai
Apache License 2.0
12.38k stars 805 forks source link

Apple Silicon Support #4

Open nicosuave opened 7 months ago

nicosuave commented 7 months ago

Awesome project. Apple Silicon support would be great to see!

danielhanchen commented 7 months ago

The more upvotes this gets will equal to the priority!! If it's a big must - I'll try my best to implement it!

danielhanchen commented 7 months ago

WAITT there's 17 upvotes for Apple Silicon support??? Oh my

ChristianWeyer commented 5 months ago

WAITT there's 17 upvotes for Apple Silicon support??? Oh my

30… 😎

danielhanchen commented 5 months ago

@ChristianWeyer Oh my 30!! Ok ok this definitely is a very good indication that people want Apple + MLX support :)) I will have to readjust my engineering roadmap :)

mbtre commented 4 months ago

1+ This would be absolutely awesome!

danielhanchen commented 4 months ago

Yeee I can see this is super popular - sadly I thought more about it - itll be a bit more complicated than I though + until I get a Mac (I use Linux + Windows!!!) it'll be a bit of an issue for me to test training runs :( I thought maybe I could just take MLX and sprinkle some Unsloth magic, but sadly that's not the case :(

vlbosch commented 2 months ago

@danielhanchen I have an M3 Max Macbook Pro with 128GB Unified RAM. Happy to help you port this great project to Apple Silicon 😃 If it's any helpful, I can spin up a VM and give you SSH access. IIRC macOS VM's on AS have direct GPU access and can run Metal with near native performance.

CyborgArmy83 commented 2 months ago

Would also love to see Apple Silicon support in the future!

danielhanchen commented 2 months ago

Ill see but sadly I'm not a Mac user :(

WiseFarAI commented 2 months ago

I seriously think that if you made a crowdfunding and posted it here and on LLAMA subreddit we could collectively get you the funds to buy a good Apple Silicon machine. Although I am wondering if it's doable at this stage from a software perspective. I am not sure how much you would have to rely on frameworks like MLX / Metal, but I do know their development is very active for the last couple of months, so it might become viable soon.

kevin801221 commented 2 months ago

Please support Apple Mac silicon M2、M3 series chips！！ It would be perfect！！！ pls！

danielhanchen commented 2 months ago

Oh my yikes ok this is ballooning!

mobguang commented 1 month ago

Waiting for the good news for supporting Apple Mac silicon!

sealad886 commented 1 month ago

Just to throw this out there: According to Apple (which notes MPS support with PyTorch version 1.12) and PyTorch (a press release dated May 2022), Apple Silicon Metal Performance Shaders (MPS) support has been standard in PyTorch for a while.

In fact, the standard pip install torch installation will do everything needed (i.e. nothing special) to enable PyTorch GPU acceleration on Apple Silicon. I personally stopped installing the nightly download months ago, once I realized that I was putting way more effort into making sure my version was up-to-date than was necessary.

To note, pip install torch installs version 2.3.0 as of this writing.

At this point it almost seems like more effort to not to support Metal. I'm sure there's complexities and reasons for it that I'm not aware of, but I wonder if there's old practices that pervade the LLM space that need to be mixed up?

danielhanchen commented 1 month ago

Oh interesting if Pytorch already has been on MPS, hmm I'll see what I can do! (Once I get a Mac :))

daniel-luper commented 1 month ago

Here's an interesting article arguing that any speedup from MLX over PyTorch's MPS is due to a small optimization mistake that could be easily fixed. The article is from a few months ago so that optimization issue might even be fixed by now: https://medium.com/@koypish/mps-or-mlx-for-domestic-ai-the-answer-will-surprise-you-df4b111de8a0

It does seem like PyTorch's MPS implementation is the move 👀, I agree with everyone above that Unsloth on Apple Silicon would be heavenly 🙏

danielhanchen commented 1 month ago

Oh interesting article! I don't disagree itll be very cool :)

MalikRumi commented 2 weeks ago

@danielhanchen Another Mac user, chugging along on my M1. So don't forget us when people talk about newer chips, please. Now as to what you need to do this work: what about renting a Mac? Here's a simple google search result: https://www.google.com/search?client=safari&rls=en&q=rent+apple+silicon&ie=UTF-8&oe=UTF-8 I note one of them is even pay as you go. You could probably raise enough money of a short term rental just from the people on this thread. Check it out, give us a budget, and let the magic happen.

danielhanchen commented 1 week ago

Sorry on the delay - just relocated to SF!

It isn't the issue of renting - it's more I'm not used to it :( I'll try getting one in the following weeks :)

abeppu commented 3 days ago

Hey, I wanted to throw out a lesser stepping-stone sort of request -- can there at a minimum be mac wheels with stubs?

For example, my team develops on macs but runs experiments and deploys models onto linux, with cuda GPUs. However, developing on mac when the unsloth package cannot be installed in a local environment means that e.g. IDEs cannot tab-complete to suggest class or method names, linters/checkers cannot flag when you misspelled a keyword argument, etc.

To me it's acceptable that to run an experiment, I need to provision a GPU instance. Even if it 'can' work on my laptop, it's likely faster and worthwhile for me to run it on a more powerful instance. But running a GPU instance just be able to install the package and discover (as a runtime error) that I accidentally passed warm_up_ratio=0.1 in UnslothTrainingArguments when actually it needs to be warmup_ratio ... seems like a waste of both human time and machine time. That's the kind of thing that a linter could have caught locally, if only I were able to install (but not run) unsloth in my mac development environment.

danielhanchen commented 2 days ago

Oh interesting - ie just unsloth as a pure python package on Mac, but temporarily disable the fast code paths?

abeppu commented 2 days ago

Yeah, so far as I'm concerned, if it throws "NotImplemented" errors, but has the class/method signatures, that would be a meaningful step forward from my point of view. Roughly, developers and CI environments would be able to set up an environment that lets them check/lint even if they can't run the full functionality. If that gives you an incremental path to eventually provide that full behavior, that's great!

On Fri, Jul 5, 2024, 20:26 Daniel Han @.***> wrote:

Oh interesting - ie just unsloth as a pure python package on Mac, but temporarily disable the fast code paths?

— Reply to this email directly, view it on GitHub https://github.com/unslothai/unsloth/issues/4#issuecomment-2211612297, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABD6MWW37OFEV5DHPFBE23ZK5PV5AVCNFSM6AAAAABADU7SJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJRGYYTEMRZG4 . You are receiving this because you commented.Message ID: @.***>