search

guoqincode / Open-AnimateAnyone

Unofficial Implementation of Animate Anyone
2.9k stars 233 forks source link

SparseCausalAttention2D #9

Closed kecol closed 9 months ago

kecol commented 9 months ago

While reading the code I saw that the standard BasicTransformerBlock from diffusers has been replaced with a modified version that utilizes a new class called SparseCausalAttention2D for the attn1 layer. Could you specify where this class is defined? Or maybe, were you able to successfully train the model without using this class (replacing it with a different one)?

guoqincode commented 9 months ago

If you do not set unet_use_cross_frame_attention=True, this class will not be used. AnimateAnyone does not use cross frame attention.

kecol commented 9 months ago

Thank you @guoqincode for your quick answer. The AnimateAnyone paper stated that all self-attention layers were replaced with spatial attention layers. I initially thought this was in reference to the aforementioned change.