State-of-the-Art Deep Learning scripts organized by models - easy to train and deploy with reproducible accuracy and performance on enterprise-grade infrastructure.
13.59k
stars
3.24k
forks
source link
[PyTorch] NGC Pytorch Image 24.02 has excessive debug logging #1391
Related to NGC PyTorch Image 24.02 (and maybe later)
Describe the bug
I'm sorry if this is not the correct place to report, I could not find any Github repo or contact address for maintainers of the NVIDIA NGC PyTorch images.
The PyTorch version in the NGC images is compiled from source (I find the choice of just using the latest commit at the time of building, rather than the latest stable release, problematic, but that is not the topic of this issue). Unfortunately, it is compiled with the flag PGNCCL_ENABLE_HASH enabled, so NCCL will print a hash for every collective operation on every rank, making all log output practically unreadable when using distributed training, especially when using deepspeed ZeRO. There is no way to suppress this log output other than recompiling PyTorch.
To Reproduce
Use nvcr.io/nvidia/pytorch:24.02-py3
Run a distributed training using NCCL
Observe excessive logging
Expected behavior
This debug logging is not necessary and should not be enabled.
Related to NGC PyTorch Image 24.02 (and maybe later)
Describe the bug I'm sorry if this is not the correct place to report, I could not find any Github repo or contact address for maintainers of the NVIDIA NGC PyTorch images.
The PyTorch version in the NGC images is compiled from source (I find the choice of just using the latest commit at the time of building, rather than the latest stable release, problematic, but that is not the topic of this issue). Unfortunately, it is compiled with the flag
PGNCCL_ENABLE_HASHenabled, so NCCL will print a hash for every collective operation on every rank, making all log output practically unreadable when using distributed training, especially when using deepspeed ZeRO. There is no way to suppress this log output other than recompiling PyTorch.To Reproduce
nvcr.io/nvidia/pytorch:24.02-py3Expected behavior This debug logging is not necessary and should not be enabled.
Environment
pytorch:24.02-py3