mlx5 connect on mlx5_1 failed: Connection timed out

shinoharakazuya commented 1 week ago

I'm running NGC's hpl benchmark test from Slurm. When I ran hpl in an hpl container on two servers with 8 GPUs per node, I encountered a UCX error.

Command line: Please see log file.
UCX version used (from github branch XX or release YY) + UCX configure flags (can be checked by ucx_info -v): Please see log file.
Any UCX environment variables used

OS version (e.g Linux distro) + CPU architecture (x86_64/aarch64/ppc64le/...): Please see log file.
- cat /etc/issue or cat /etc/redhat-release + uname -a
- For Nvidia Bluefield SmartNIC include cat /etc/mlnx-release (the string identifies software and firmware setup)
For RDMA/IB/RoCE related issues: Please see log file.
- Driver version:
  - rpm -q rdma-core or rpm -q libibverbs
  - or: MLNX_OFED version ofed_info -s
- HW information from ibstat or ibv_devinfo -vv command
For GPU related issues:
- GPU type : H100
- Cuda:
  - Drivers version:12.2
  - Check if peer-direct is loaded: lsmod|grep nv_peer_mem and/or gdrcopy: lsmod|grep gdrdrv : Please see log file.

OpenMPI version:5.0.3
Output of ucx_info -d to show transports and devices recognized by UCX: Please see log file.

shinoharakazuya commented 1 week ago

yosefe commented 6 days ago

@shinoharakazuya can you pls post the output of show_gids command, and check if setting UCX_IB_ROCE_LOCAL_SUBNET=y helps to resolve the issue?

changchengx commented 5 days ago

@jandres742 FYI

yosefe commented 5 days ago

NOTE: This issue happens on Nvidia internal cluster

openucx / ucx