search

state-spaces / mamba

Mamba SSM architecture
Apache License 2.0
12.44k stars 1.05k forks source link

support training on near-infinite sequence lengths? #441

Open luke-mcdermott-mi opened 2 months ago

luke-mcdermott-mi commented 2 months ago

infinite sequence length inference is currently supported by stepping through each token; however, I do not see any support for training on this. The parallel scan applied per batch does not have support to save the final hidden state for the following batch, so are there any plans to train mamba on context lengths > 4096 or whatever is the length cap from memory?

JulienSiems commented 2 months ago

There has been a recent paper on this: https://arxiv.org/pdf/2406.02080