search

QUVA-Lab / e2cnn

E(2)-Equivariant CNNs Library for Pytorch
https://quva-lab.github.io/e2cnn/
Other
599 stars 74 forks source link

equivariant Transformer #49

Closed liminyu-web closed 2 years ago

liminyu-web commented 3 years ago

hi~i want to ask if i can build rotated-equivariant swin transformer/Transformer anchitectural in E2CNN library? thank you!!!

Gabri95 commented 3 years ago

Hi @liminyu-web

Our library does not implement equivariant attention but I think we provide all the group-theory/equivariance related tools you need to implement one yourself. I am aware of a few equivariant transformer works, but do you have some particular paper in mind you would like to replicate? If you can share some papers I can take a look to check that the library actually includes all you need.

Best, Gabriele

liminyu-web commented 3 years ago

Thank you very much for your reply！！ I read the paper ReDet: A Rotation-equivariant Detector for Aerial Object Detection ，their work uses your modules e2cnn to implement ReResNet-50 and ReFPN，so i want to ask if i can do similar work on Swin Transformer，i have tested that Swin Transformer can get higher accuracy than ReResNet-50 in rotated object detection. I haven‘t found related work to equivariant transformer（self-attention），do you have any recommendations~~thank you！

Thanks， Minyu

Gabri95 commented 3 years ago

Hi @liminyu-web

These are a few works on equivariant attentions that I know (not necessarily an exhaustive list): Lie Transformers, SE(3) Transformers, Equivariant Stand-Alone Self-Attention.

Maybe you can check the SE(3) Transformers: although they consider the group of 3D rotations (SO(3)), the method can be adapted for 2D rotations (SO(2)). To implement such equivariant attention, you still need to parameterize an equivariant convolutional kernel. Indeed, they use Eq.8 inside Eq. 10 and 11 but Eq. 8 just describes SO(3) steerable filters. You could try to use our implementation of SO(2) steerable filters to build a similar architecture which, instead, is SO(2) equivariant.

I would be very courious to know if this would work, let me know if you try it

Best, Gabriele

liminyu-web commented 3 years ago

Thank you very much for your reply~~ I will read these papers and try it, if it would work , i will reply here~

Thanks, Minyu

Gabri95 commented 2 years ago

Hey!

Since there have been no more comments, I am closing this issue. Please, feel free to reopen it or contact me if you have more questions :)

Best, Gabriele