Inconsistent Results Despite Setting Random Seed

[Moreland-cas]

Hello,

I am encountering an issue with inconsistent results when running the code, despite setting the random seed for reproducibility. I have tried using the original random seed setting code provided in your repository, as well as an adjusted version to ensure deterministic behavior, especially when using PyTorch. However, I am still facing variable outcomes in repeated runs.

The original seed setting code in your repository is:

def set_random_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)

To enhance the consistency, I modified the seed setting function as follows:

def set_random_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    # torch.cuda.manual_seed(seed)
    # torch.cuda.manual_seed_all(0) 
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    # torch.use_deterministic_algorithms(True)

Despite these changes, the results are still inconsistent across different runs. I am using a single GPU for these experiments. I suspect there might be other parts of the code or dependencies that introduce randomness.

Could you please help me identify any potential sources of variability or suggest further modifications to ensure consistent results?

Thank you for your assistance.

[Moreland-cas]

torch.use_deterministic_algorithms(True)

torch.use_deterministic_algorithms(True) solves this issue, plus setting "export CUBLAS_WORKSPACE_CONFIG=:4096:8"