search

state-spaces / mamba

Mamba SSM architecture
Apache License 2.0
13.28k stars 1.13k forks source link

Mamba-S4 #73

Open turian opened 11 months ago

turian commented 11 months ago

Could you share code for Mamba-S4, the audio model for longform signals?

albertfgu commented 11 months ago

We don't have that code in a releasable form. It's essentially the S4 model put inside the Mamba block, so it's pretty easy to implement.

ftgreat commented 10 months ago

@albertfgu one question to bother you.

Is there the minor module of S4 in mamba or s4 repos? Thanks.

albertfgu commented 10 months ago

We aren't using submodules. You'll need to incorporate the code from the S4 repo separately.

btrude commented 10 months ago

@albertfgu Thank you for your work on mamba. I was wondering if you could elaborate on the architecture of the S4+mamba block a bit more? I am trying to reproduce mamba+s4 with a unet as described in the paper, but I so far only have working forward pass for a sashimi-like model with just mamba blocks. Any additional detail you can provide would be much appreciated!

albertfgu commented 10 months ago

Take the Mamba block and replace the S6 layer with S4.

hungdche commented 6 months ago

@albertfgu To my understanding, Mamba-S4 is this block with the S4Block as mixer. Is that correct?