search

flux-framework / flux-core

core services for the Flux resource management framework
GNU Lesser General Public License v3.0
167 stars 50 forks source link

cray MPI: MPIDI_OFI_mpi_init_hook:Invalid argument #5976

Open garlick opened 4 months ago

garlick commented 4 months ago

Problem: multi-node MPI hello world on slurm-launched flux on frontier fails with

$ flux run --env=LD_LIBRARY_PATH=$(dirname $(flux config builtin pmi_library_path)):$LD_LIBRARY_PATH -n2 -N2 ./mpi_hello
0.160s: job.exception type=exec severity=0 MPI_Abort: Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(170).......:  MPID_Init(501)..............:  MPIDI_OFI_mpi_init_hook(599): OFI fi_open domain failed (ofi_init.c:599:MPIDI_OFI_mpi_init_hook:Invalid argument)
0.160s: job.exception type=exec severity=0 MPI_Abort: Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(170).......:  MPID_Init(501)..............:  MPIDI_OFI_mpi_init_hook(599): OFI fi_open domain failed (ofi_init.c:599:MPIDI_OFI_mpi_init_hook:Invalid argument)
flux-job: task(s) exited with exit code 143
MPICH ERROR [Rank 0] [job id fYSTCFMZH] [Thu May 16 17:41:46 2024] [frontier08676] - Abort(1091983) (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPICH ERROR [Rank 0] [job id fYSTCFMZH] [Thu May 16 17:41:46 2024] [frontier08677] - MPIR_Init_thread(170).......:
MPID_Init(501)..............:
Abort(1091983) (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIDI_OFI_mpi_init_hook(599): OFI fi_open domain failed (ofi_init.c:599:MPIDI_OFI_mpi_init_hook:Invalid argument)

MPIR_Init_thread(170).......:
MPID_Init(501)..............:
MPIDI_OFI_mpi_init_hook(599): OFI fi_open domain failed (ofi_init.c:599:MPIDI_OFI_mpi_init_hook:Invalid argument)

Single node works OK.

This is on

commands:               0.55.0
libflux-core:           0.55.0
broker:                 0.55.0
FLUX_URI:               ssh://frontier05043/tmp/flux-Bp2MUb/local-0
build-options:          +hwloc==2.8.0+zmq==4.3.5
garlick commented 4 months ago

We tried with -o verbose=2 and verified that we got a Flux pmi server trace, so MPI is definitely using Flux PMI.

We also tried setting FI_PROVIDER=tcp,shm and it still fails in about the same place but the errors are a little different

MPICH ERROR [Rank 0] [job id fV8HYkfZ] [Thu May 16 19:30:06 2024] [frontier00034] - Abort(1616271) (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(170).......:
MPID_Init(501)..............:
MPIDI_OFI_mpi_init_hook(573):
open_fabric(1434)...........: OFI fi_getinfo() failed (ofi_init.c:1434:open_fabric:No data available)