Add SeaFormer model

[inderpreetsingh01]

Model description

The computational cost and memory requirement render many computer vision models unsuitable on the mobile device, especially for the high-resolution per-pixel semantic segmentation task. SeaFormer (Squeeze-enhanced Axial Transformer) designed a generic attention block characterized by the formulation of squeeze Axial and detail enhancement. Coupled with a light segmentation head, they achieve the best trade-off between segmentation accuracy and latency on the ARM-based mobile devices on the ADE20K and Cityscapes datasets. They beat both the mobile-friendly rivals and Transformer-based counterparts with better performance and lower latency.

Open source status

• [X] The model implementation is available
• [X] The model weights are available

Provide useful links for the implementation

Paper: https://arxiv.org/pdf/2301.13156.pdf Code and weights: https://github.com/fudan-zvg/SeaFormer Authors: @wwqq @lzrobots @speedinghzl

cc: @NielsRogge @alaradirik

[strankid]

Hi. I would like to work on this.

[alaradirik]

Hi @inderpreetsingh01 thanks for opening the issue, SeaFormer definitely seems like a good addition to the library!

Are you planning to work on this model? If not, @strankid could start working on it or you two could collaborate on a PR. In either case, you could take a look at our model addition guidelines, as well as the transformers code of other segmentation models such as SegFormer, MaskFormer, Mask2Former and OneFormer.

[lzrobots]

hello, fancy to make SETR on board？

[strankid]

@alaradirik should I begin with a WIP PR?

[inderpreetsingh01]

Hi @alaradirik thanks for sharing the resources. I will be working on adding this model. @strankid if you want we can collaborate on this. @lzrobots i saw both SeaFormer and SETR use mmseg, we can look into it.

[strankid]

@inderpreetsingh01 i'm down to collaborate!

[alaradirik]

Great :) @strankid @inderpreetsingh01 you can ping me if you have questions about the library or need help with anything (e.g. model conversion).

It'd be great if you could open a WIP PR, as it'd make it easier to ask / answer questions and do a preliminary review later down the road.

[inderpreetsingh01]

thanks @alaradirik will do it. @strankid can you share your mail id so that we can connect on slack?

[strankid]

@inderpreetsingh01 my email is apoorv96@gmail.com. Should I create the PR or would you like to?

[inderpreetsingh01]

@strankid sure you can create the pr.

[strankid]

@inderpreetsingh01 Saw you created the wip pr. Since you have my email, just contact me and let me know how you want to split the work.