search

FlagOpen / FlagScale

FlagScale is a large model toolkit based on open-sourced projects.
Other
167 stars 42 forks source link

[NewFeature] Add Unified Sequence Parallelism #156

Closed zhaoyinglia closed 2 months ago

zhaoyinglia commented 3 months ago

Implement refer to:

Jiarui Fang and Shangchun Zhao. 2024. USP: A Unified Sequence Parallelism Approach for Long Context Generative AI. https://doi.org/10.48550/arXiv.2405.07719 image

Final version: https://github.com/FlagOpen/FlagScale/pull/187