Add context parallel for fleet

PaddlePaddle / Paddle

PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice （『飞桨』核心框架，深度学习&机器学习高性能单机、分布式训练和跨平台部署）

http://www.paddlepaddle.org/

Apache License 2.0

21.66k stars 5.44k forks source link

Add context parallel for fleet #64087

Open zhangyuqin1998 opened 1 week ago

zhangyuqin1998 commented 1 week ago

PR Category

Distributed Strategy

PR Types

New features

Description

为fleet新增cp的支持。cp是和sep类似的在序列维度的并行，和mp、dp、pp是正交的，但不能和sep一起使用。cp使用ring flash attention方法来实现序列维度的并行。

paddle-bot[bot] commented 1 week ago

你的PR提交成功，感谢你对开源项目的贡献! 请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。 Your PR has been submitted. Thanks for your contribution! Please wait for the result of CI firstly. See Paddle CI Manual for details.

CLAassistant commented 1 week ago

All committers have signed the CLA.

CLAassistant commented 1 week ago

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

zhangyuqin1998 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

paddle-ci-bot[bot] commented 3 days ago

Sorry to inform you that 7576f77's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.