search

unslothai / unsloth

Finetune Llama 3.2, Mistral, Phi, Qwen 2.5 & Gemma LLMs 2-5x faster with 80% less memory
https://unsloth.ai
Apache License 2.0
18.66k stars 1.31k forks source link

[Feature Request] DDP #127

Open nivibilla opened 10 months ago

nivibilla commented 10 months ago

Wanted to make an issue for this instead of constantly asking in discord.

I saw the other ticket for multigpu fp16 training which is also nice. But ddp would let users scale up training that can happen on single gpus to multi gpu for linear speedup.

danielhanchen commented 10 months ago

@nivibilla We're actively working to prep for a later release to most likely provide DDP to the OSS :) We're still figuring out licensing and distribution methods, so just figuring out those stuff first :)