search

Westlake-AI / MogaNet

[ICLR 2024] MogaNet: Efficient Multi-order Gated Aggregation Network
https://arxiv.org/abs/2211.03295
Apache License 2.0
162 stars 13 forks source link

Is channel aggregation block stronger than SE block? #2

Closed xuesongnie closed 1 year ago

xuesongnie commented 1 year ago

Does replacing SE with CA in any network have an increase of 0.6?

Lupin1998 commented 1 year ago

Hi, @xuesongnie, thanks for your question. Unfortunately, we haven't applied the proposed CA to other architectures except for MogaNet in our experiments. But, I believe that CA will achieve performance gains when applied to the FFN module (e.g., in ViTs and MetaFormer variants), and it is likely to outperform SE under the same parameter budget. Meanwhile, CA doesn't require the extra Norm layer compared to SE, which might be easier to adopt in ViTs. If you are interested in CA, maybe you can try it in your experiments, and we can discuss it in a specific scenario. Please free to ask me or contact me at WeChat (Lupin_1998) if you have further questions.

Lupin1998 commented 1 year ago

I closed this issue if there is no more question. You can reopen it or start a new issue if you have more questions. Thanks again for your attention.