flx42
commented
2 years ago This might be a bit tricky to debug, but we can try :)
MOFED kernel version
What's the version of MLNX OFED on the system? (kernel side). A command like this one might help:
$ dpkg -l | grep mlnx-ofed-kernel
ii mlnx-ofed-kernel-dkms 5.3-OFED.5.3.1.0.0.1 all DKMS support for mlnx-ofed kernel modules
ii mlnx-ofed-kernel-utils 5.3-OFED.5.3.1.0.0.1 amd64 Userspace tools to restart and tune mlnx-ofed kernel modulesInfiniBand devices
NCCL INFO NET/IB : Using [0]mlx5_1:1/IB/SHARP [1]mlx5_3:1/IB/SHARP [2]mlx5_6:1/IB/SHARP [3]mlx5_8:1/IB/SHARP [4]mlx5_4:1/RoCE [5]mlx5_10:1/RoCE ; OOB enp226s0:10.16.2.21<0>I'm not sure if that's the root cause, but it might be a problem to have the mlx5_4 and mlx5_10 devices exposed here, particularly in RoCE mode.
If you have ENROOT_RESTRICT_DEV y in enroot.conf, you can exclude those devices by running your container with MELLANOX_VISIBLE_DEVICES=1,3,6,8. Verify that this line only shows IB devices after this change.
nccl-tests
Can you install https://github.com/NVIDIA/nccl-tests.git inside the Dockerfile and try with the all_reduce_perf binary?
Here is my Dockerfile recipe:
RUN cd /usr/local/src && \
NCCL_TESTS_VERSION="1f8f5416863a3082975b10eaa05fecee6fe870c8" && \
curl --proto '=https' -fSsL https://github.com/NVIDIA/nccl-tests/archive/${NCCL_TESTS_VERSION}.tar.gz | tar xz && \
cd nccl-tests-${NCCL_TESTS_VERSION} && \
make MPI=1 && \
install -m 755 build/all_* build/broadcast_* build/reduce_* /usr/local/bin Then to launch it:
$ srun --container-image=<image> --mpi=pmix --ntasks-per-node=8 all_reduce_perf -b 4 -e 4G -f 2 -c 1 -n 100I suspect this will also fail, and that would be a good step since TF would be out of the picture.
andrew-johnson-melb
This is really a combined problem with slurm + tensorflow + pyxis. I'm yet to hear anything from the TF team, so I was hoping you might have an idea @flx42 (any suggestions would be very much appreciated).
Something weird happens with NCCL when inside an enroot container submitted via slurm with pyxis. Essentially, the tf.distribute.MirroredStrategy strategy fails, due to NCCL errors. I've provided all the information below.
Again, I understand this is not entirely appropriate for this repo. But any help would be great. This issue is at the boundary between a few things.
System information
The distributed training runs fails when training via slurm (using srun).
The code is run inside an enroot container. Due to slurm this container has a number of slurm specific environment variables set.
So, using
MirroredStrategyto distribute training fails due to NCCL errors on a simple example.Note, a number of other distributed options work as is highlighted in the code.
NOTE, this code works fine outside of the slurm environment (in the exact same container). The slurm environment variables seem to be creating an issue with NCCL.
The srun command looks like
Error
Error when using 21.07 container and tf-nightly
Function call stack: train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function