[FEATURE]: Support SP+PP in Llama etc.

hpcaitech / ColossalAI

Making large AI models cheaper, faster and more accessible

https://www.colossalai.org

Apache License 2.0

38.28k stars 4.3k forks source link

[FEATURE]: Support SP+PP in Llama etc. #5866

Open GuangyaoZhang opened 3 days ago

GuangyaoZhang commented 3 days ago

Describe the feature

Currently most models like Llama does not support SP together with PP. Please add support for this.

Issues-translate-bot commented 3 days ago

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Title: [FEATURE]: Support SP+PP in Llama etc.