Open kihangyoun opened 3 years ago
Hi @kihangyoun,
/opt/local/mpi/2021.2.0/etc/tuning_icx_shm-ofi_mlx.dat
?Hi @brminich, Thanks for your attention.
Were you able to get lower start-up times with any other than UCX transport? For now I'd attribute this slowness to MPI_Init implementation and the collectives being used inside it
Hello All,
There is a problem that takes a lot of time during MPI startup(MPI_INIT) with a large number of MPI ranks. The section that takes time is: ucp_worker.c:1719 UCX INFO ep_cfg[0], ep_cfg[2] When I use 76,000 MPI rank, it take 68~72 seconds in MPI startup(MPI_INIT). Here is printed log and bold font is bottleneck(I guess).
*Intel MPI version is 2021.2.0, UCX is 1.10.0 & MOFED 5.2-1.0.4.0. Please let me know if you have any suggestion or need any additional information. Thanks