search

xdit-project / xDiT

xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) on multi-GPU Clusters
Apache License 2.0
480 stars 40 forks source link

Why Latte Model can‘t support PipeFusion #225

Open philipwan opened 3 weeks ago

philipwan commented 3 weeks ago

Open-Sora-v1.1’s Transformer module come from Latte, I see PipeFusion doesn't support Latte in README. How does xDit support Open-Sora-v1.1/1.2 module, any suggestions ?

Thank you~

feifeibear commented 3 weeks ago

We just have no time to support pipefuson on Latte. Currently, we are focusing on CogVideoX, which seems more popular right now. I found ColossalAI, the framework developed by the company behind Open-Sora, has already used pipefusion.

https://github.com/hpcaitech/ColossalAI/blob/7cf9df07bcb267c0839d8880b109c4c7d55e80fa/colossalai/inference/modeling/layers/distrifusion.py#L3

Would like to help us on supporting PipeFusion for Latte-based models?

philipwan commented 2 weeks ago

We just have no time to support pipefuson on Latte. Currently, we are focusing on CogVideoX, which seems more popular right now. I found ColossalAI, the framework developed by the company behind Open-Sora, has already used pipefusion.

https://github.com/hpcaitech/ColossalAI/blob/7cf9df07bcb267c0839d8880b109c4c7d55e80fa/colossalai/inference/modeling/layers/distrifusion.py#L3

Would like to help us on supporting PipeFusion for Latte-based models?

Sorry for my mistake, I want to support for open-sora-plan v1.2 which has same struction with CogVideoX. Open-Sora may be slightly worse video quality， but ColossalAI's PipeFusion is a good example. I will try to keep following up.

feifeibear commented 2 weeks ago

We just have no time to support pipefuson on Latte. Currently, we are focusing on CogVideoX, which seems more popular right now. I found ColossalAI, the framework developed by the company behind Open-Sora, has already used pipefusion. https://github.com/hpcaitech/ColossalAI/blob/7cf9df07bcb267c0839d8880b109c4c7d55e80fa/colossalai/inference/modeling/layers/distrifusion.py#L3 Would like to help us on supporting PipeFusion for Latte-based models?

Sorry for my mistake, I want to support for open-sora-plan v1.2 which has same struction with CogVideoX. Open-Sora may be slightly worse video quality， but ColossalAI's PipeFusion is a good example. I will try to keep following up.

We can work together on open-sora-plan v1.2.