search

xindongzhang / ELAN

[ECCV2022] Efficient Long-Range Attention Network for Image Super-resolution
Apache License 2.0
212 stars 19 forks source link

Positional Encoding #3

Open mrluin opened 2 years ago

mrluin commented 2 years ago

Hello,

Thanks for your great work, an efficient and neat transformer framework is essential for low-level vision I think.

According to your work, I tried discard the attention mask and positional encoding in SwinIR, the training and inference speed is largely improved, and the attention mask has slight effect on performance. However, the performance severely droped after removing RPE in original SwinIR.

Could you please give me some hints about how can we discard RPE (and attention mask) correctly? directly removing codes related to positional encoding or removing positional encoding should incorporate someother necessary elements?

Looking forward to your reply, thanks~

bonlime commented 2 years ago

It probably works for authors because they add Shift-Conv (which is almost a conv3x3), default SwinIR doesn't have any 3x3 convs so postitional embedding is required