garlick
commented
1 year ago Greetings! For a start you might try getting a smaller allocation and running flux with a non-interactive "initial program" (what comes after flux start) and some PMI debugging:
$ FLUX_PMI_DEBUG=1 mpirun flux start /bin/trueYou might see some output similar to (this is for 2 nodes):
boot_pmi: flux-pmi-client: trying 'simple'
boot_pmi: flux-pmi-client: trying 'simple'
boot_pmi: simple: selected
boot_pmi: simple: selected
boot_pmi: simple: initialize: rank=1 size=2 name=-: success
boot_pmi: simple: initialize: rank=0 size=2 name=-: success
boot_pmi: simple: get key=flux.instance-level: invalid key argument
boot_pmi: simple: get key=flux.instance-level: invalid key argument
boot_pmi: simple: get key=flux.taskmap: invalid key argument
boot_pmi: simple: get key=flux.taskmap: invalid key argument
boot_pmi: simple: get key=PMI_process_mapping value=(vector,(0,1,2)): success
boot_pmi: simple: get key=PMI_process_mapping value=(vector,(0,1,2)): success
boot_pmi: simple: put key=1 value={"hostname":"system76-pc","pubkey":"sa)Zs}@7U}9*0iukx0F(SS$@fQt7b]zX/>ICATg=","uri":""}: success
boot_pmi: simple: put key=0 value={"hostname":"system76-pc","pubkey":"Q.1u>Twr>676eV)f3y>5UJJ#t6#B2PuVg#+X{rYA","uri":"ipc:///tmp/flux-i9vMHd/tbon-0"}: success
boot_pmi: simple: barrier: success
boot_pmi: simple: barrier: success
boot_pmi: simple: get key=1 value={"hostname":"system76-pc","pubkey":"sa)Zs}@7U}9*0iukx0F(SS$@fQt7b]zX/>ICATg=","uri":""}: success
boot_pmi: simple: get key=0 value={"hostname":"system76-pc","pubkey":"Q.1u>Twr>676eV)f3y>5UJJ#t6#B2PuVg#+X{rYA","uri":"ipc:///tmp/flux-i9vMHd/tbon-0"}: success
boot_pmi: simple: get key=0 value={"hostname":"system76-pc","pubkey":"Q.1u>Twr>676eV)f3y>5UJJ#t6#B2PuVg#+X{rYA","uri":"ipc:///tmp/flux-i9vMHd/tbon-0"}: success
boot_pmi: simple: get key=0 value={"hostname":"system76-pc","pubkey":"Q.1u>Twr>676eV)f3y>5UJJ#t6#B2PuVg#+X{rYA","uri":"ipc:///tmp/flux-i9vMHd/tbon-0"}: success
boot_pmi: simple: get key=1 value={"hostname":"system76-pc","pubkey":"sa)Zs}@7U}9*0iukx0F(SS$@fQt7b]zX/>ICATg=","uri":""}: success
boot_pmi: simple: get key=1 value={"hostname":"system76-pc","pubkey":"sa)Zs}@7U}9*0iukx0F(SS$@fQt7b]zX/>ICATg=","uri":""}: success
boot_pmi: simple: barrier: success
boot_pmi: simple: barrier: success
boot_pmi: simple: finalize: success
boot_pmi: simple: finalize: successOr you might see that it can't find a PMI method that works and falls back to "single" in which case all your flux brokers think they are rank 0, and sadness results :sob:
My guess is that your openmpi module is only offering "pmix" as a bootstrap method which flux does not support natively. There is a flux-pmix add-on which you could pursue if that is the only option, but maybe you have mvapich or mpich installed and could load that module instead? mpich's mpirun offers plain old simple pmi and should be able to bootstrap flux.
If we can get flux to parallel bootstrap then maybe we can move on to figuring out how to get an interactive flux shell to work?
ligerzero-ai
grondo
Configuration - conda-installed flux
on slurm I can just run:
srun --mpi=pmi2 --pty flux start {{command}}to get flux to see all nodes
Is there any equivalent command for a PBSPro queueing system?
I've tried:
mpiexec flux startandmpirun flux startwith no real luck: