search

Westlake-AI / MogaNet

[ICLR 2024] MogaNet: Efficient Multi-order Gated Aggregation Network
https://arxiv.org/abs/2211.03295
Apache License 2.0
162 stars 13 forks source link

CA module #22

Open Zhangyuhaoo opened 1 month ago

Zhangyuhaoo commented 1 month ago

Sorry to bother you. Hello, I have read your paper and found it very impressive. I have a small question: can I use the CA module you proposed to replace the FFN layer in ViT? Again, I apologize for my interruption and look forward to your reply and suggestions. Thank you!

Lupin1998 commented 2 weeks ago

Hello, @Zhangyuhaoo. Sorry for the late reply. The proposed CA module in MogaNet enhances the MixFFN, which inserts a DWConv3x3 in FFN (proposed in PVTv2), which can be regarded as the GRN block proposed in ConvNeXtv2. You could replace the FFN block with CA+MixFNN in ViTs to enhance the overall performance, i.e., adding the CA module and the DWConv3x3 in the FFN block. Hoping this message is not too late for you to be helpful, and feel free to ask me if there are more questions.