Open yxchng opened 1 year ago
Hi,
Yes it should be supported, you would have to create the bias Tensor though, and pass it as attn_bias
in the memory_efficient_attention
function.
@danthe3rd
attn = (
attn.view(B, q_h, q_w, k_h, k_w) + rel_h[:, :, :, :, None] + rel_w[:, :, :, None, :]
).view(B, q_h * q_w, k_h * k_w)
the shape of attn needs to be changed here. how can i do that for memory efficient attention?
@danthe3rd
attn = ( attn.view(B, q_h, q_w, k_h, k_w) + rel_h[:, :, :, :, None] + rel_w[:, :, :, None, :] ).view(B, q_h * q_w, k_h * k_w)
the shape of attn needs to be changed here. how can i do that for memory efficient attention?
Try this
rel_h = rel_h.unsqueeze(4).repeat(1, 1, 1, 1, k_w).view(B, q_h * q_w, k_h * k_w)
rel_w = rel_w.unsqueeze(3).repeat(1, 1, 1, k_h, 1).view(B, q_h * q_w, k_h * k_w)
bias = rel_h + rel_w
hello,Have you solved this problem so far?
Segment Anything uses an attention that has relative position embedding like this
how can i use xformers efficient attention for this? thanks.