Open carns opened 2 years ago
Actually this may be related to FI_UNIVERSE_SIZE, which I did not set. I just realized that it stops at 256 by default. I'll retry this scenario with a larger size.
The problem persists with FI_UNIVERSE_SIZE set to 2048, which would be sufficient for all of the server and client processes in this test case.
I'll have to try this again later with debugging symbols enabled, but I'm pretty consistently hitting this:
... when running the ssg-benchmarking example on Summit with this configuration (I'm not sure if the client portion of the test is a factor or if the server group encounters this on its own):
jsrun -l cpu-cpu -a 1 -b none -r 20 -n 320 -c 2 ${BIN_DIR}/ssg-launch-group -s $((600)) -f ssg-bench.out -n scale-test verbs://mlx5_0 mpi