search

hpcaitech / ColossalAI

Making large AI models cheaper, faster and more accessible
https://www.colossalai.org
Apache License 2.0
38.28k stars 4.3k forks source link

[FEATURE]: LoRA with sharded model #5826

Open KaiLv69 opened 1 week ago

KaiLv69 commented 1 week ago

Describe the feature

Hi, when training big model like llama2-70b with lora, it will run into oom due to the unsharded model.

It could help a lot if lora supported with GeminiPlugin or HybridParallelPlugin. Wonder if any plan to support that?