search

huggingface / accelerate

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support
https://huggingface.co/docs/accelerate
Apache License 2.0
7.77k stars 941 forks source link

Optimize the megatron plugin #2822

Closed zhangsheng377 closed 3 months ago

zhangsheng377 commented 4 months ago

Optimize the megatron plugin, and make the megatron's version to r0.5.0. https://github.com/huggingface/accelerate/pull/2501#issuecomment-2126931526 https://github.com/huggingface/accelerate/issues/2390#issuecomment-2136834517

zhangsheng377 commented 3 months ago

@muellerzr @pacman100

muellerzr commented 3 months ago

Hey @zhangsheng377, let me give it a look now. FYI Sourab is no longer a part of HF, so I will be taking up Megatron/DS/FSDP

HuggingFaceDocBuilderDev commented 3 months ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

zhangsheng377 commented 3 months ago

Hey @zhangsheng377, let me give it a look now. FYI Sourab is no longer a part of HF, so I will be taking up Megatron/DS/FSDP

It's really unfortunate to hear about Sourab's departure. I will try to modify the code format shortly. Thank you for letting me know.

zhangsheng377 commented 3 months ago

@muellerzr Is there anything else that needs to be done? And by the way, what are the requirements to become a committer?

zhangsheng377 commented 3 months ago

Thanks we can go ahead and merge this. Why do you want permissions to be a committer out of curiosity?

Thank you. Because I have a strong interest in open source, I think the Accelerate project is very promising. And my current working is training large models with Megatron, so I think I can contribute to Accelerate. (By the way, I also contributed distributed LoRa features to PEFT 😃)

muellerzr commented 3 months ago

For security reasons, you can only be at most a contributor, and we need to manually approve running the CI. However you have contributed to this project and it will show up in github as such :)