search

OpenNMT / OpenNMT-py

Open Source Neural Machine Translation and (Large) Language Models in PyTorch
https://opennmt.net/
MIT License
6.7k stars 2.25k forks source link

set random seed for a multi-GPU model #2516

Open Galaxy-Husky opened 9 months ago

Galaxy-Husky commented 9 months ago

Hi, I have a question about how to set seed when using multi gpus.

The following code snippet uses torch.cuda.manul_seed funciton.

if is_cuda and seed > 0:
        # These ensure same initialization in multi gpu mode
        torch.cuda.manual_seed(seed)

But I found a warning in the document of torch.cuda.manual_seed, which says "If you are working with a multi-GPU model, this function is insufficient to get determinism. To seed all GPUs, use manual_seed_all()."

Would it be better to change torch.cuda.manual_seed(seed) to torch.cuda.manual_seed_all(seed) ?

vince62s commented 9 months ago

Thanks. Have you tried to run twice the same run? not getting the same results in multi gpus ?