search

HazyResearch / m2

Repo for "Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture"
Apache License 2.0
538 stars 44 forks source link

What category does the M2 model belong to #34

Open 41924076 opened 5 months ago

41924076 commented 5 months ago

Hello, thank you for your great work! M2bert paper mentioned that "Monarch Mixer is part of a new class of architectures called state-space models (SSMs), which include S4, Mamba, and BiGS". Is Monarch Mixer and M2BERT a part of SSMs? I consider M2BERT to be: (1) replace attention with bidirectional gated convolutions with a residual convolution, and set the Monarch matrices to DFT and inverse DFT matrices to speed up DFT for conv; (2)In the dimension mixer, replace the two dense matrices in MLPs with learned block-diagonal matrices to speed up MLP computation.

I wonder which part of it is related to SSM? I would be very grateful if you could help me with the answer : )

DanFu09 commented 5 months ago

Great question! Every convolution is an SSM so that’s what we mean by SSM model. The dimension mixer is orthogonal.

On Wed, May 29, 2024 at 12:41 AM 41924076 @.***> wrote:

Hello, thank you for your great work! M2bert paper mentioned that "Monarch Mixer is part of a new class of architectures called state-space models (SSMs), which include S4, Mamba, and BiGS". Is Monarch Mixer and M2BERT a part of SSMs? I consider M2BERT to be: (1) replace attention with bidirectional gated convolutions with a residual convolution, and set the Monarch matrices to DFT and inverse DFT matrices to speed up DFT for conv; (2)In the dimension mixer, replace the two dense matrices in MLPs with learned block-diagonal matrices to speed up MLP computation.

I wonder which part of it is related to SSM? I would be very grateful if you could help me with the answer : )

— Reply to this email directly, view it on GitHub https://github.com/HazyResearch/m2/issues/34, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDDIIURHEI5SUWOUK35W53ZEWBBNAVCNFSM6AAAAABIOLK2QGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGMZDENRQGMZDKOA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

41924076 commented 5 months ago

Thank you so much for your answer! Can every convolution is every model be considered as SSM or only what kind of convolution can be considered as SSM? And how to understand the difference between MAMBA and M2 models in terms of SSM?