search

PKU-YuanGroup / Open-Sora-Plan

This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.
MIT License
11.25k stars 1k forks source link

Is DeepSpeed-Ulysses the sequence parallel method used in Open-Sora-Plan v1.2.0? #376

Open TheTinyTeddy opened 1 month ago

TheTinyTeddy commented 1 month ago

Hi, thank you for the great work!

I was wondering is DeepSpeed-Ulysses the sequence parallel method used in both inference/training of Open-Sora-Plan v1.2.0?

(As a side note, I think you could take a look at the hybrid ring/ulysses method featured in https://github.com/feifeibear/long-context-attention/tree/main )

apprivoiser commented 3 weeks ago

We only used the sp method in the inference of Open-Sora-Plan v1.2.0. Thank you for your recommendation, we will pay attention to this work.