search

hpcaitech / ColossalAI

Making large AI models cheaper, faster and more accessible
https://www.colossalai.org
Apache License 2.0
38.68k stars 4.34k forks source link

[FEATURE]: Add Ulysses Sequence Parallelism support for Command-R, Qwen2 and ChatGLM #5853

Closed GuangyaoZhang closed 2 months ago

GuangyaoZhang commented 3 months ago

Describe the feature

Please add Ulysses Sequence Parallelism support for Command-R, Qwen2 and ChatGLM