search

PKU-YuanGroup / Open-Sora-Plan

This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.
MIT License
11.55k stars 1.03k forks source link

[feat] Use Ulysses/FastSeq for QKV computation in DiT/Latte/MMDiT #131

Open kabachuha opened 8 months ago

kabachuha commented 8 months ago

--Borrowed from OpenDiT--

FastSeq boosts performance thanks for improved attention memory communication time

https://github.com/NUS-HPC-AI-Lab/OpenDiT?tab=readme-ov-file#fastseq

Example diff implementation on top of existing DiT/Latte

https://github.com/NUS-HPC-AI-Lab/OpenDiT/pull/92/commits/d7be4992cbe028c7068f35e8463f9ce1003aabb1

Utility functions reference:

https://github.com/kabachuha/OpenMMDiT/blob/d7be4992cbe028c7068f35e8463f9ce1003aabb1/opendit/utils/operation.py#L1

kabachuha commented 8 months ago

This PR has it as a part of OpenDiT borrowing https://github.com/PKU-YuanGroup/Open-Sora-Plan/pull/73

LinB203 commented 8 months ago

I will check it.