search

state-spaces / mamba

Mamba SSM architecture
Apache License 2.0
11.55k stars 946 forks source link

Mamba-S4 #73

Open turian opened 6 months ago

turian commented 6 months ago

Could you share code for Mamba-S4, the audio model for longform signals?

albertfgu commented 6 months ago

We don't have that code in a releasable form. It's essentially the S4 model put inside the Mamba block, so it's pretty easy to implement.

ftgreat commented 6 months ago

@albertfgu one question to bother you.

Is there the minor module of S4 in mamba or s4 repos? Thanks.

albertfgu commented 6 months ago

We aren't using submodules. You'll need to incorporate the code from the S4 repo separately.

btrude commented 5 months ago

@albertfgu Thank you for your work on mamba. I was wondering if you could elaborate on the architecture of the S4+mamba block a bit more? I am trying to reproduce mamba+s4 with a unet as described in the paper, but I so far only have working forward pass for a sashimi-like model with just mamba blocks. Any additional detail you can provide would be much appreciated!

albertfgu commented 5 months ago

Take the Mamba block and replace the S6 layer with S4.

hungdche commented 2 months ago

@albertfgu To my understanding, Mamba-S4 is this block with the S4Block as mixer. Is that correct?