Closed toandd2 closed 2 months ago
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
This issue was closed because it has been inactive for 7 days since being marked as stale.
We are using
examples/nlp/language_modeling/megatron_gpt_pretraining.py
with a small GPT model (1.35 billion parameters) on 4 H100 DGX nodes (8 GPUs each). Our DGX nodes are connected using InfiniBand. The training starts but hangs in the middle (around 197 steps) without any exception. When it stops, the power consumption drops to approximately 100W/700W (500-690W is normal), and the GPU utilization remains at 100%.We have enabled the NCCL debug flag and embedded the log file in this issue.
We solved this issue by adding a custom synchronization in the training step below this line:
We also enabled
export NCCL_P2P_DISABLE=1
. However, this hotfix has slowed down our training.The training step duration increased from 1.08 seconds per iteration to 1.25 seconds per iteration.
Steps/Code to reproduce bug
Config file:
Expected behavior
The training start normally and run compute with faster speed per iteration without needed to
export NCCL_P2P_DISABLE=1
.Environment overview (please complete the following information)
Environment details
Using NVIDIA docker image.
Additional context NCCL logs with debug flag:
nccl_logs.zip GPU:
NVIDIA H100 80GB HBM3 x32
InfiniBand:MT4129