Closed xuesongnie closed 1 year ago
Hi, @xuesongnie, thanks for your question. Unfortunately, we haven't applied the proposed CA to other architectures except for MogaNet in our experiments. But, I believe that CA will achieve performance gains when applied to the FFN module (e.g., in ViTs and MetaFormer variants), and it is likely to outperform SE under the same parameter budget. Meanwhile, CA doesn't require the extra Norm layer compared to SE, which might be easier to adopt in ViTs. If you are interested in CA, maybe you can try it in your experiments, and we can discuss it in a specific scenario. Please free to ask me or contact me at WeChat (Lupin_1998) if you have further questions.
I closed this issue if there is no more question. You can reopen it or start a new issue if you have more questions. Thanks again for your attention.
Does replacing SE with CA in any network have an increase of 0.6?