search

openpmix / pmi-shim

Shim library that exposes PMI-1 and PMI-2 interfaces, translating such calls to PMIx
BSD 3-Clause "New" or "Revised" License
0 stars 3 forks source link

PMI2 shim crashes #4

Open jjhursey opened 5 years ago

jjhursey commented 5 years ago

Background information

What version of the PMIx Reference Library are you using? (e.g., v1.0, v2.1, git master @ hash, etc.)

Describe how PMIx was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

PMIx built with:

    ./configure --prefix=${PMIX_ROOT} \
                --with-hwloc=${HWLOC_INSTALL_PATH} \
                --with-libevent=${LIBEVENT_INSTALL_PATH} \
                --enable-debug

Open MPI built with:

    ./configure --prefix=${MPI_ROOT} \
                --with-hwloc=${HWLOC_INSTALL_PATH} \
                --with-libevent=${LIBEVENT_INSTALL_PATH} \
                --with-pmix=${PMIX_ROOT} \
                --enable-mpirun-prefix-by-default

Please describe the system on which you are running

Details of the problem

I was testing PMI2 shim with the pmi2_client.c in the v3.1.4 release.

[mpiuser@3cd5ac47d23f test]$ env | grep PMIX
PMIX_ROOT=/home/mpiuser/local/pmix
[mpiuser@3cd5ac47d23f test]$ mpicc pmi2_client.c -I${PMIX_ROOT}/include -L${PMIX_ROOT}/lib -lpmi2 -Wall -g -O0 -o pmi2_client

If I don't specify a gds component it will crash with the stack below:

Program trace with no gds specified ``` [mpiuser@3cd5ac47d23f test]$ mpirun -np 2 ./pmi2_client INFO: spawned=0 size=20 rank=0 appnum=0 INFO: jobid=2406219777 INFO: TI1 : PASS INFO: TI2 : PASS ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed) INFO: TI3 : PASS ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed) INFO: TI4 : PASS INFO: spawned=0 size=20 rank=1 appnum=0 INFO: jobid=2406219777 INFO: TI1 : PASS INFO: TI2 : PASS ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed) INFO: TI3 : PASS ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed) INFO: TI4 : PASS INFO: tkey=sharedFilename tval=pmix-pmi2-check val= found=0 INFO: tkey=sharedFilename tval=pmix-pmi2-check val= found=0 FATAL: PMI2_Info_GetNodeAttr does not find expected key at test_item5:341 FATAL: value does not meet expectation at test_item5:342 FATAL: value does not meet expectation at test_item5:343 INFO: TI5 : FAIL FATAL: PMI2_Info_GetNodeAttr does not find expected key at test_item5:341 FATAL: value does not meet expectation at test_item5:342 FATAL: value does not meet expectation at test_item5:343 INFO: TI5 : FAIL INFO: TI6 : PASS INFO: TI6 : PASS INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7 INFO: TI7 : PASS INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7 INFO: TI7 : PASS INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7 INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7 INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7 INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7 [3cd5ac47d23f:00138] [[36716,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217 [3cd5ac47d23f:00144] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334 FATAL: PMI2_KVS_Get [KEY-2=?] -1 INFO: TI8 : FAIL INFO: Rank 1 executing Put of key KEY-1 [3cd5ac47d23f:00143] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334 FATAL: PMI2_KVS_Get [KEY-2=?] -1 INFO: TI8 : FAIL INFO: Rank 0 executing Put of key KEY-0 INFO: Rank 1 executing Fence INFO: Rank 0 executing Fence INFO: Rank 0 executing Get of key KEY-0 INFO: Rank 1 executing Get of key KEY-0 INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7 INFO: Rank 0 executing Get of key KEY-1 INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7 INFO: Rank 1 executing Get of key KEY-1 INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7 INFO: Rank 0 executing Get of key KEY-2 INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7 INFO: Rank 1 executing Get of key KEY-2 FATAL: PMI2_KVS_Get [KEY-2=?] -1 INFO: TI9 : FAIL INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=72 FATAL: PMI2_KVS_Get [KEY-2=?] -1 INFO: TI9 : FAIL INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=72 INFO: 1 : PMI2_KVS_Put [RANK1-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 [3cd5ac47d23f:00138] [[36716,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 [3cd5ac47d23f:00144] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334 FATAL: 1 : PMI2_KVS_Get [RANK2-key-0=?] -1 INFO: TI10 : FAIL INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0 [3cd5ac47d23f:00138] [[36716,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217 [3cd5ac47d23f:00143] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334 FATAL: 0 : PMI2_KVS_Get [RANK2-key-0=?] -1 INFO: TI10 : FAIL -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[36716,1],1] Exit code: 4 -------------------------------------------------------------------------- ```

I noticed a slightly different behavior if I forced the hash component.

Program trace with `hash` gds specified ``` [mpiuser@3cd5ac47d23f test]$ export PMIX_MCA_gds=hash [mpiuser@3cd5ac47d23f test]$ mpirun -np 2 ./pmi2_client INFO: spawned=0 size=20 rank=0 appnum=0 INFO: jobid=2406678529 INFO: TI1 : PASS INFO: TI2 : PASS ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed) INFO: TI3 : PASS ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed) INFO: TI4 : PASS INFO: tkey=sharedFilename tval=pmix-pmi2-check val=pmix-pmi2-check found=1 INFO: TI5 : PASS INFO: TI6 : PASS INFO: spawned=0 size=20 rank=1 appnum=0 INFO: jobid=2406678529 INFO: TI1 : PASS INFO: TI2 : PASS ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed) INFO: TI3 : PASS ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed) INFO: TI4 : PASS INFO: tkey=sharedFilename tval=pmix-pmi2-check val=pmix-pmi2-check found=1 INFO: TI5 : PASS INFO: TI6 : PASS INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7 INFO: TI7 : PASS INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7 INFO: TI7 : PASS INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7 INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7 INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7 INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7 [3cd5ac47d23f:00149] [[36723,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217 [3cd5ac47d23f:00155] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334 FATAL: PMI2_KVS_Get [KEY-2=?] -1 INFO: TI8 : FAIL INFO: Rank 1 executing Put of key KEY-1 [3cd5ac47d23f:00154] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334 FATAL: PMI2_KVS_Get [KEY-2=?] -1 INFO: TI8 : FAIL INFO: Rank 0 executing Put of key KEY-0 INFO: Rank 1 executing Fence INFO: Rank 0 executing Fence INFO: Rank 1 executing Get of key KEY-0 INFO: Rank 0 executing Get of key KEY-0 INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7 INFO: Rank 1 executing Get of key KEY-1 INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7 INFO: Rank 0 executing Get of key KEY-1 INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7 INFO: Rank 1 executing Get of key KEY-2 INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7 INFO: Rank 0 executing Get of key KEY-2 FATAL: PMI2_KVS_Get [KEY-2=?] -1 INFO: TI9 : FAIL INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=15 INFO: 1 : PMI2_KVS_Put [RANK1-key-0=aaaaaaaaaaaaaaa] 0 FATAL: PMI2_KVS_Get [KEY-2=?] -1 INFO: TI9 : FAIL INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=15 INFO: 0 : PMI2_KVS_Put [RANK0-key-0=aaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-1=bbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-2=ccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-3=ddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-4=eeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-1=bbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-5=fffffffffffffff] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-2=ccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-6=ggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-3=ddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-7=hhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-4=eeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-8=iiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-5=fffffffffffffff] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-9=jjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-6=ggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-10=kkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-7=hhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-11=lllllllllllllll] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-8=iiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-9=jjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-12=mmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-10=kkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-13=nnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-11=lllllllllllllll] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-14=ooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-12=mmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-15=ppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-16=qqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-13=nnnnnnnnnnnnnnn] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-17=rrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-14=ooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-18=sssssssssssssss] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-15=ppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-19=ttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-16=qqqqqqqqqqqqqqq] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-20=uuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-17=rrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-21=vvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-18=sssssssssssssss] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-22=wwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-19=ttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-23=xxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-20=uuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-24=yyyyyyyyyyyyyyy] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-21=vvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-25=zzzzzzzzzzzzzzz] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-22=wwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-23=xxxxxxxxxxxxxxx] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-26=aaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-24=yyyyyyyyyyyyyyy] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-27=bbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-25=zzzzzzzzzzzzzzz] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-28=ccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-26=aaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-29=ddddddddddddddd] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-27=bbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-30=eeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-28=ccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-31=fffffffffffffff] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-29=ddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-32=ggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-30=eeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-33=hhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-31=fffffffffffffff] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-34=iiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-32=ggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-35=jjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-33=hhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-36=kkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-34=iiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-37=lllllllllllllll] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-35=jjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-38=mmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-36=kkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-39=nnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-37=lllllllllllllll] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-40=ooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-38=mmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-41=ppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-39=nnnnnnnnnnnnnnn] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-42=qqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-40=ooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-43=rrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-41=ppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-42=qqqqqqqqqqqqqqq] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-44=sssssssssssssss] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-43=rrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-45=ttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-44=sssssssssssssss] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-46=uuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-45=ttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-47=vvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-46=uuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-48=wwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-47=vvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Put [RANK1-key-49=xxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-48=wwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Put [RANK0-key-49=xxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-0=aaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-1=bbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-2=ccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-3=ddddddddddddddd] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-4=eeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-5=fffffffffffffff] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-6=ggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-7=hhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-8=iiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-9=jjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-10=kkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-0=aaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-11=lllllllllllllll] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-1=bbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-12=mmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-2=ccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-13=nnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-3=ddddddddddddddd] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-14=ooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-4=eeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-15=ppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-5=fffffffffffffff] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-16=qqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-17=rrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-6=ggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-18=sssssssssssssss] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-7=hhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-19=ttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-8=iiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-20=uuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-9=jjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-21=vvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-10=kkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-22=wwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-11=lllllllllllllll] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-23=xxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-12=mmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-24=yyyyyyyyyyyyyyy] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-13=nnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-25=zzzzzzzzzzzzzzz] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-14=ooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-26=aaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-15=ppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-27=bbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-16=qqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-28=ccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-17=rrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-18=sssssssssssssss] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-29=ddddddddddddddd] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-19=ttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-30=eeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-31=fffffffffffffff] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-20=uuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-32=ggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-21=vvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-33=hhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-22=wwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-34=iiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-23=xxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-24=yyyyyyyyyyyyyyy] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-35=jjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-36=kkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-25=zzzzzzzzzzzzzzz] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-37=lllllllllllllll] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-26=aaaaaaaaaaaaaaa] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-38=mmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-27=bbbbbbbbbbbbbbb] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-39=nnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-28=ccccccccccccccc] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-40=ooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-29=ddddddddddddddd] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-41=ppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-30=eeeeeeeeeeeeeee] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-42=qqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-31=fffffffffffffff] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-43=rrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-32=ggggggggggggggg] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-33=hhhhhhhhhhhhhhh] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-44=sssssssssssssss] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-45=ttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-34=iiiiiiiiiiiiiii] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-46=uuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-35=jjjjjjjjjjjjjjj] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-47=vvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-36=kkkkkkkkkkkkkkk] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-48=wwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-37=lllllllllllllll] 0 INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-49=xxxxxxxxxxxxxxx] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-38=mmmmmmmmmmmmmmm] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-39=nnnnnnnnnnnnnnn] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-40=ooooooooooooooo] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-41=ppppppppppppppp] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-42=qqqqqqqqqqqqqqq] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-43=rrrrrrrrrrrrrrr] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-44=sssssssssssssss] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-45=ttttttttttttttt] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-46=uuuuuuuuuuuuuuu] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-47=vvvvvvvvvvvvvvv] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-48=wwwwwwwwwwwwwww] 0 INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-49=xxxxxxxxxxxxxxx] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-0=aaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-0=aaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-1=bbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-1=bbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-2=ccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-2=ccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-3=ddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-3=ddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-4=eeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-4=eeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-5=fffffffffffffff] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-5=fffffffffffffff] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-6=ggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-6=ggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-7=hhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-7=hhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-8=iiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-8=iiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-9=jjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-9=jjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-10=kkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-10=kkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-11=lllllllllllllll] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-11=lllllllllllllll] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-12=mmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-12=mmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-13=nnnnnnnnnnnnnnn] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-13=nnnnnnnnnnnnnnn] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-14=ooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-14=ooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-15=ppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-15=ppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-16=qqqqqqqqqqqqqqq] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-16=qqqqqqqqqqqqqqq] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-17=rrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-17=rrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-18=sssssssssssssss] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-18=sssssssssssssss] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-19=ttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-19=ttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-20=uuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-20=uuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-21=vvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-21=vvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-22=wwwwwwwwwwwwwww] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-22=wwwwwwwwwwwwwww] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-23=xxxxxxxxxxxxxxx] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-23=xxxxxxxxxxxxxxx] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-24=yyyyyyyyyyyyyyy] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-24=yyyyyyyyyyyyyyy] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-25=zzzzzzzzzzzzzzz] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-25=zzzzzzzzzzzzzzz] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-26=aaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-26=aaaaaaaaaaaaaaa] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-27=bbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-27=bbbbbbbbbbbbbbb] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-28=ccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-28=ccccccccccccccc] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-29=ddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-29=ddddddddddddddd] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-30=eeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-30=eeeeeeeeeeeeeee] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-31=fffffffffffffff] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-31=fffffffffffffff] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-32=ggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-32=ggggggggggggggg] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-33=hhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-33=hhhhhhhhhhhhhhh] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-34=iiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-34=iiiiiiiiiiiiiii] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-35=jjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-35=jjjjjjjjjjjjjjj] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-36=kkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-37=lllllllllllllll] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-36=kkkkkkkkkkkkkkk] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-38=mmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-37=lllllllllllllll] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-39=nnnnnnnnnnnnnnn] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-38=mmmmmmmmmmmmmmm] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-40=ooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-39=nnnnnnnnnnnnnnn] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-40=ooooooooooooooo] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-41=ppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-41=ppppppppppppppp] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-42=qqqqqqqqqqqqqqq] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-42=qqqqqqqqqqqqqqq] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-43=rrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-43=rrrrrrrrrrrrrrr] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-44=sssssssssssssss] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-45=ttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-44=sssssssssssssss] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-46=uuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-45=ttttttttttttttt] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-47=vvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-46=uuuuuuuuuuuuuuu] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-47=vvvvvvvvvvvvvvv] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-48=wwwwwwwwwwwwwww] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-48=wwwwwwwwwwwwwww] 0 INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-49=xxxxxxxxxxxxxxx] 0 INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-49=xxxxxxxxxxxxxxx] 0 [3cd5ac47d23f:00149] [[36723,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217 [3cd5ac47d23f:00154] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334 FATAL: 0 : PMI2_KVS_Get [RANK2-key-0=?] -1 INFO: TI10 : FAIL [3cd5ac47d23f:00155] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334 FATAL: 1 : PMI2_KVS_Get [RANK2-key-0=?] -1 INFO: TI10 : FAIL -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[36723,1],0] Exit code: 3 -------------------------------------------------------------------------- ```

I'm not sure if this is a PMIx library issue or something with Open MPI as I see a mix of ORTE and PMIx error messages in there.

rhc54 commented 4 years ago

This should/will be shifted to the new PMI-1/2 backward compatibility repo: https://github.com/openpmix/pmi-shim