search

Vchitect / Latte

Latte: Latent Diffusion Transformer for Video Generation.
Apache License 2.0
1.44k stars 147 forks source link

Option to use ReBased Linear Attention and RingAttention #51

Open kabachuha opened 4 months ago

kabachuha commented 4 months ago

Not tested yet, as it needs training runs, but I think it can be helpful for you

ReBased repo: https://github.com/corl-team/rebased

Ring Attention implementation by Lucidrains https://github.com/lucidrains/ring-attention-pytorch

maxin-cn commented 4 months ago

Not tested yet, as it needs training runs, but I think it can be helpful for you

ReBased repo: https://github.com/corl-team/rebased

Thanks for your PR, we will check it.

bhack commented 3 months ago

Be aware that there could be some problem in the case you want to compile it https://github.com/pytorch/pytorch/issues/121386

KimmiShi commented 3 months ago

ring attention is not the most efficient method for this model.