search

state-spaces / mamba

Mamba SSM architecture
Apache License 2.0
13.03k stars 1.11k forks source link

The Reproducibility of Mamba #493

Open Yisen-Feng opened 3 months ago

Yisen-Feng commented 3 months ago

Hi,

Thank you for your awesome work! I have a query regarding the reproducibility of Mamba. Even though I use torch.use_deterministic_algorithms(True, warn_only=False) in my code, after adding Mamba, my code lost reproducibility. I wonder if it is possible to have a deterministic Mamba.

GuHongyang commented 2 months ago

I also have met this problem!

tridao commented 2 months ago

The current backward pass is not deterministic (it uses atomic adds).

tridao commented 1 month ago

I think that's the pytorch cumsum, so it's out of the scope of this repo.

NLHuang commented 1 month ago

I think that's the pytorch cumsum, so it's out of the scope of this repo.

I see. Thank you.