Seeding not compatible with Monai

[lukas-folle-snkeos]

Bug description

Using random monai transformations as pre-processing for the dataset, introduces randomness that - at the moment- is not controlled by lightining's seed_everything.

Rather on the transformations or the monai.transforms.Compose one has to call set_random_state.

I am not sure, if this is something lightning would like to handle, but I just wanted to make you aware of it.

What version are you seeing the problem on?

v2.2

How to reproduce the bug

import lightning as L
import numpy as np
from monai.transforms import RandGaussianNoised

data = {"image": np.array([10, 10, 10])}
transform = RandGaussianNoised(["image"], prob=1.0)

L.seed_everything(42)
result_a = transform(data)["image"]

L.seed_everything(42)
result_b = transform(data)["image"]

np.testing.assert_array_equal(result_a, result_b)

Error messages and logs

AssertionError: 
    Arrays are not equal

Fix

import lightning as L
import numpy as np
from monai.transforms import RandGaussianNoised

data = {"image": np.array([10, 10, 10])}
transform = RandGaussianNoised(["image"], prob=1.0)

L.seed_everything(42)
# ⬇️
transform.set_random_state(42)
result_a = transform(data)["image"]

L.seed_everything(42)
# ⬇️
transform.set_random_state(42)
result_b = transform(data)["image"]

np.testing.assert_array_equal(result_a, result_b)

pass

cc @awaelchli

[awaelchli]

Hey @lukas-folle-snkeos

Thanks for asking.

These functions are internally creating a random state, essentially like so:

state = np.random.RandomState()

(found if you follow the trail here)

This means "create a new random state that is completely independent of the global numpy state". Since this state is not seeded, it will be completely random. Proof:

import numpy as np

np.random.seed(10)
state = np.random.RandomState()
print("from unseeded state", state.randint(0, 100, size=5))
print("from global seeded state", np.random.randint(0, 100, size=5))

Run this snippet multiple times and observe that the numbers generated from the global state don't change, but the ones from the unseeded state do.

You can now replace np.random.seed(10) above with L.seed_everything(10) and observe the exact same behavior. The above represents what is happening in your transform.

Since the transform is using an internal local random state, Lightning cannot and should not interfere with that. I recommend no action to take here. The L.seed_everything is meant for seeding the global state. The "everything" in the name refers to "every library" (torch, numpy, python) but not in the literal sense everything.

Hope this explanation helps.

[lukas-folle-snkeos]

Hey @awaelchli,

Thanks, then I understood it the same way as you did! I am not sure about the number of people using both, lightning and monai, but do you think it would make sense to either mention this case in the docs or check the combination of the packages during runtime?

[awaelchli]

Personally I would prefer if Lightning stayed neutral in the sense that it just follows PyTorch as close as possible. Your initial concern about the seed not taking effect could also be one directed to PyTorch (e.g. torch.manual_seed).For example, in case you would have used raw PyTorch instead of Lightning to start with. Then, you would have gone on the same path to discover that calling transform.set_random_state(42) was a must. But the reality is, Monai's default is what it is, to detach itself from the global state means it will not be deterministic unless you call its dedicated functions to be so. And this I interpret to be a deliberate design choice by the makers of Monai. If Lightning would print warnings or infos (I assume that's what you meant by "check the combination of the packages during runtime"), then there is a possibility for this to be noise or false positives. I'm not a Monai user so this last part I'm not sure.

[lukas-folle-snkeos]

Ok, thanks for giving your perspective! I'll consider opening an issue with Monai instead.