jsrun on IBM Sierra doesn't bootstrap flux instance

[dongahn]

I installed a recent version of flux and tried to bootstrap an instance using jsrun at 2 nodes. Somehow, one of the brokers exited with an error during startup:

sierra4361{dahn}21: bsub -nnodes 2 -Is -XF -G guests /usr/bin/tcsh
sierra4368{dahn}34: jsrun -a 1 -c 40 -g 4 -n 2 /usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/default/bin/flux start -o,-S,log-filename=out

bsub allocated 2 nodes: sierra2985 and sierra2986.

The first broker is still running on sierra2985.

sierra2985{dahn}23: ps x
   PID TTY      STAT   TIME COMMAND
 21818 ?        Sl     0:00 /usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/flux-0.9.0-20180621/libexec/flux/cmd/flux-broker -S log
 22160 ?        S      0:00 /bin/tcsh
 24243 ?        S      0:00 sshd: dahn@pts/0
 24244 pts/0    Ss     0:00 -tcsh
 26427 pts/0    R+     0:00 ps x
102835 ?        Sl     0:00 /opt/ibm/spectrum_mpi/jsm_pmix/bin/jsmd csm=52694 sierra4368 --peer -ptsargs -p 192.168.81.26,40137 -r 10
sierra2985{dahn}24:

But the second broker exited -- not on sierra2986

sierra2986{dahn}30: ps x
   PID TTY      STAT   TIME COMMAND
 28359 ?        S      0:00 sshd: dahn@pts/0
 28360 pts/0    Ss     0:00 -tcsh
 33611 pts/0    R+     0:00 ps x
103179 ?        Sl     0:00 /opt/ibm/spectrum_mpi/jsm_pmix/bin/jsmd csm=52694 sierra4368 --peer -ptsargs -p 192.168.81.26,40137 -r 10
sier

@trws said he actually used MPICH Hydra to launch Flux on Sierra. I can also consider doing this but this would be pretty involved for end users to use.

[trws]

Yeah, jsrun wont bootstrap flux right now any more than flux will bootstrap spectrum (without a lot of work). I’m not sure what’s broken there, but OpenMPI mpirun works IIRC, so the spectrum mpirun might (I have not checked).

On 26 Jun 2018, at 12:16, Dong H. Ahn wrote:

I installed a recent version of flux and tried to bootstrap an instance using jsrun at 2 nodes. Somehow, one of the brokers exited with an error during startup:

sierra4361{dahn}21: bsub -nnodes 2 -Is -XF -G guests /usr/bin/tcsh
sierra4368{dahn}34: jsrun -a 1 -c 40 -g 4 -n 2 
/usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/default/bin/flux 
start -o,-S,log-filename=out

bsub allocated 2 nodes: sierra2985 and sierra2986.

The first broker is still running on sierra2985.

sierra2985{dahn}23: ps x
   PID TTY      STAT   TIME COMMAND
 21818 ?        Sl     0:00 
/usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/flux-0.9.0-20180621/libexec/flux/cmd/flux-broker 
-S log
 22160 ?        S      0:00 /bin/tcsh
 24243 ?        S      0:00 sshd: dahn@pts/0
 24244 pts/0    Ss     0:00 -tcsh
 26427 pts/0    R+     0:00 ps x
102835 ?        Sl     0:00 /opt/ibm/spectrum_mpi/jsm_pmix/bin/jsmd 
csm=52694 sierra4368 --peer -ptsargs -p 192.168.81.26,40137 -r 10
sierra2985{dahn}24:

But the second broker exited -- not on sierra2986

sierra2986{dahn}30: ps x
   PID TTY      STAT   TIME COMMAND
 28359 ?        S      0:00 sshd: dahn@pts/0
 28360 pts/0    Ss     0:00 -tcsh
 33611 pts/0    R+     0:00 ps x
103179 ?        Sl     0:00 /opt/ibm/spectrum_mpi/jsm_pmix/bin/jsmd 
csm=52694 sierra4368 --peer -ptsargs -p 192.168.81.26,40137 -r 10
sier

@trws said he actually used MPICH Hydra to launch Flux on Sierra. I can also consider doing this but this would be pretty involved for end users to use.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/flux-framework/flux-core/issues/1555

[dongahn]

We can ask this to IBM control system guys today as well then.

[dongahn]

IBM guys doesn't know exactly why this fails. The recommendation was to try with mpirun to see if that's a little better and also to involve Ralph Castain about PMIX's PMI backward compatible mode. They say JSM doesn't support PMI at all but PMIX supports this so PMIX is the right layer to look at.

My hope is IBM will set up a slack channel where Spectrum MPI and JSM issues like this can be discussed. That way, we can get the responses sooner, rather than later.

[dongahn]

OK. I've found a few issues that explain why jsrun couldn't bootstrap flux.

1. jsrun doesn't export PMIX_SERVER_URI but PMIX_SERVER_URI2 instead. Whereas, flux only looks for PMIX_SERVER_URI in here

2. Even if I changed the name to PMIX_SERVER_URI2 and add the library directory where you can find the Spectrum-bundled libpmix.so to LD_LIBRARY_PATH, flux still fails to resolve PMI_Init. Apparently, PMIX bundled with IBM Spectrum doesn't have PMI symbols defined.

@rhc54: are you supposed to have PMI function definitions within the PMIX library to provide backward compatibility? If so, do you know why spectrum MPI provided one doesn't have it?

I will also contact IBM developers on this as well.

[rhc54]

Let me address both of those issues. First, JSM is based on the PMIx v2.x release series where we have both TCP and usock connection support. PMIX_SERVER_URI provides the rendezvous information for the usock channel and is what you would see from a PMIx v1.x server (which doesn't have TCP support). PMIX_SERVER_URI2 is for the TCP channel.

FWICT JSM disables the usock channel by default, and so you don't see PMIX_SERVER_URI in the environment. If you just change the name of the envar, it likely won't work depending upon the PMIx that flux is built against - the usock channel from PMIx v1.2 won't understand the value.

The backward compatibility symbols are in a separate pair of libraries and not in libpmix itself. This was done to avoid problems for those software packages that tried to build statically against libpmix as well as against someone else's PMI-1/2 libraries (e.g., OpenMPI packagers). We also made building libpmi and libpmi2 optional - again, so packagers wouldn't create conflicts with other libraries.

If you aren't seeing libpmi and libpmi2, my guess is that IBM has chosen not to enable building of the PMI-1/2 libraries because they don't support those use-cases. Given the problems, you may find it simpler to just add PMIx support to flux. MPICH now supports PMIx, so I'm not sure what you gain by sticking with the older libraries, and it would allow you to smoothly move between JSM and flux.

[rhc54]

Just a little further clarification. The libpmi and libpmi2 libraries provided by PMIx will not intercommunicate with the flux-provided PMI libraries. Understand that there never was a defined "standard" protocol underneath those definitions, and so the "libpmi" from one source (e.g., PMIx) is not expected to be able to communicate to the "libpmi" from another.

In other words, assume we have a client that is linked against the PMI library from PMIx. The PMIx library actually converts PMI calls to their PMIx equivalent and then uses the PMIx protocol to talk to the expected PMIx local server. This is what we mean by "backward compatible" - i.e., a client can contain PMI calls and still interact with a PMIx server without modifying their code. However, they won't be able to talk to a flux PMI server as the two don't speak the same language.

There are limits to the PMIx backward support. We covered the basic APIs but didn't try to cover everything as most libraries don't use some of them. Ditto for the PMI2 definitions.

[garlick]

Thanks @rhc54, that's reasonable and matches our expectations. Flux internally uses PMI_* (v1) library calls to bootstrap itself when it is not the native resource manager. Making that work under IBM's jsrun/LSF launcher is what this issue is about, as opposed to enabling spectrum MPI executables to be launched by Flux, which is #1382.

At the moment Flux's PMI client uses environmental hints to dlopen the correct PMI library to bootstrap (similar to OpenMPI schizo, as I understand it) and one of them was if PMIX_SERVER_URI is set, it dlopens libpmix.so and expects to be able to dlsym the PMI v1 symbols there. If that is no longer the right thing to do then maybe we just need to help Flux find the IBM libpmi.so instead.

[rhc54]

I see - so you want Flux to dlopen the PMIx libpmi(v1) lib so you can talk to JSM? That should work assuming we aren't missing some API you need.

If IBM isn't producing that library, note that you can always just grab a copy of PMIx v2.1.1 and build it yourself. The configure option to output the compatibility libs is --enable-pmi-backward-compatibility, though my configure help line indicates we may leave it on by default.

[garlick]

Yes if that works, it's probably the shortest path to getting Flux to run on this system. Thanks for the tip on rebuilding, in case we need that.

[dongahn]

Currently PMIx configure fails on Sierra for some reason. Would it be reasonable to extract the pmi libraries from this version and buold it against thr PMIX bundled with Spectrum?

[garlick]

I was able to configure 2.2.1 OK on sierra after creating a missing /var/tmp/garlick (!) but build fails here:

$ configure --enable-embedded-libevent
$ make
[snip]
  CC       client/pmix_client.lo
client/pmix_client.c:53:35: fatal error: event2/thread.h: No such file or directory
 #include PMIX_EVENT2_THREAD_HEADER
                                   ^
compilation terminated.
make[2]: *** [client/pmix_client.lo] Error 1
make[2]: Leaving directory `/g/g0/garlick/pmix-2.1.1/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/g/g0/garlick/pmix-2.1.1/src'
make: *** [all-recursive] Error 1
$

I'm sure we can work through this but it wouldn't hurt to ask IBM to provide the libraries in future releases of spectrum.

[dongahn]

Ok. Using the 2.2.1 release version and libevent-2.1.8-stable.tar.gz side installation, PMIX configure, make and make install worked.

lrwxrwxrwx 1 dahn dahn      21 Jun 27 10:21 libevent-2.1.so.6 -> libevent-2.1.so.6.0.2
-rwxr-xr-x 1 dahn dahn 1489568 Jun 27 10:21 libevent-2.1.so.6.0.2
-rw-r--r-- 1 dahn dahn 2519944 Jun 27 10:21 libevent.a
-rwxr-xr-x 1 dahn dahn     972 Jun 27 10:21 libevent.la
lrwxrwxrwx 1 dahn dahn      21 Jun 27 10:21 libevent.so -> libevent-2.1.so.6.0.2
lrwxrwxrwx 1 dahn dahn      26 Jun 27 10:21 libevent_core-2.1.so.6 -> libevent_core-2.1.so.6.0.2
-rwxr-xr-x 1 dahn dahn  975904 Jun 27 10:21 libevent_core-2.1.so.6.0.2
-rw-r--r-- 1 dahn dahn 1660350 Jun 27 10:21 libevent_core.a
-rwxr-xr-x 1 dahn dahn    1007 Jun 27 10:21 libevent_core.la
lrwxrwxrwx 1 dahn dahn      26 Jun 27 10:21 libevent_core.so -> libevent_core-2.1.so.6.0.2
lrwxrwxrwx 1 dahn dahn      27 Jun 27 10:21 libevent_extra-2.1.so.6 -> libevent_extra-2.1.so.6.0.2
-rwxr-xr-x 1 dahn dahn  597496 Jun 27 10:21 libevent_extra-2.1.so.6.0.2
-rw-r--r-- 1 dahn dahn  859666 Jun 27 10:21 libevent_extra.a
-rwxr-xr-x 1 dahn dahn    1014 Jun 27 10:21 libevent_extra.la
lrwxrwxrwx 1 dahn dahn      27 Jun 27 10:21 libevent_extra.so -> libevent_extra-2.1.so.6.0.2
lrwxrwxrwx 1 dahn dahn      29 Jun 27 10:21 libevent_openssl-2.1.so.6 -> libevent_openssl-2.1.so.6.0.2
-rwxr-xr-x 1 dahn dahn  166648 Jun 27 10:21 libevent_openssl-2.1.so.6.0.2
-rw-r--r-- 1 dahn dahn  175818 Jun 27 10:21 libevent_openssl.a
-rwxr-xr-x 1 dahn dahn    1034 Jun 27 10:21 libevent_openssl.la
lrwxrwxrwx 1 dahn dahn      29 Jun 27 10:21 libevent_openssl.so -> libevent_openssl-2.1.so.6.0.2
lrwxrwxrwx 1 dahn dahn      30 Jun 27 10:21 libevent_pthreads-2.1.so.6 -> libevent_pthreads-2.1.so.6.0.2
-rwxr-xr-x 1 dahn dahn   82912 Jun 27 10:21 libevent_pthreads-2.1.so.6.0.2
-rw-r--r-- 1 dahn dahn   24614 Jun 27 10:21 libevent_pthreads.a
-rwxr-xr-x 1 dahn dahn    1035 Jun 27 10:21 libevent_pthreads.la
lrwxrwxrwx 1 dahn dahn      30 Jun 27 10:21 libevent_pthreads.so -> libevent_pthreads-2.1.so.6.0.2
-rwxr-xr-x 1 dahn dahn    1088 Jun 27 10:28 libpmi.la
lrwxrwxrwx 1 dahn dahn      15 Jun 27 10:28 libpmi.so -> libpmi.so.1.0.0
lrwxrwxrwx 1 dahn dahn      15 Jun 27 10:28 libpmi.so.1 -> libpmi.so.1.0.0
-rwxr-xr-x 1 dahn dahn  881160 Jun 27 10:28 libpmi.so.1.0.0
-rwxr-xr-x 1 dahn dahn    1094 Jun 27 10:28 libpmi2.la
lrwxrwxrwx 1 dahn dahn      16 Jun 27 10:28 libpmi2.so -> libpmi2.so.1.0.0
lrwxrwxrwx 1 dahn dahn      16 Jun 27 10:28 libpmi2.so.1 -> libpmi2.so.1.0.0
-rwxr-xr-x 1 dahn dahn  880768 Jun 27 10:28 libpmi2.so.1.0.0
-rwxr-xr-x 1 dahn dahn    1096 Jun 27 10:28 libpmix.la
lrwxrwxrwx 1 dahn dahn      17 Jun 27 10:28 libpmix.so -> libpmix.so.2.1.11
lrwxrwxrwx 1 dahn dahn      17 Jun 27 10:28 libpmix.so.2 -> libpmix.so.2.1.11
-rwxr-xr-x 1 dahn dahn  813592 Jun 27 10:28 libpmix.so.2.1.11
drwx------ 2 dahn dahn    4096 Jun 27 10:21 pkgconfig
drwx------ 2 dahn dahn    4096 Jun 27 10:28 pmix

I will see if I hook this up with Flux to launch an instance with jsrun. Now that I typed up, this should have belonged to #1555.

[garlick]

Nice! I think if you make sure PMI_LIBRARY env variable points to this library, Flux will try to use it.

(Make sure PMI_FD and PMX_SERVER_URI are not set though)

[dongahn]

I made two-line changes to flux:

• Use PMIX_SERVER_URI2 instead of PMIX_SERVER_URI in here

• Replace libpmix.so with libpmi.so in here

Then, I set LD_LIBRARY_PATH to the local path where I have a newly built libpmi.so:

sierra4367{dahn}50: echo $LD_LIBRARY_PATH
/opt/ibm/spectrumcomputing/lsf/10.1/linux3.10-glibc2.17-ppc64le-csm/lib:/usr/tce/packages/cuda/cuda-9.2.88/lib64:/g/g0/dahn/workspace/PMIX-PMI/lib

But it still won't bootstrap flux with the following error:

sierra4367{dahn}47: env FLUX_PMI_DEBUG=1 jsrun -a 1 -c 40 -n2 /usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/default/bin/flux start -o,-S,log-filename=out
PMI_Init: PMIX_SERVER_URI2 is set, use PMIX's libpmi.so
PMI_Init: PMIX_SERVER_URI2 is set, use PMIX's libpmi.so
pmi_wrap_create: using /g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    no-plugins
from the file:
    help-pmix-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
-1: PMI_Init rc=1 PMI not initialized
flux-broker: PMI_Init: PMI not initialized
flux-broker: bootstrap failed
pmi_wrap_create: using /g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    no-plugins
from the file:
    help-pmix-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
-1: PMI_Init rc=1 PMI not initialized
flux-broker: PMI_Init: PMI not initialized
flux-broker: bootstrap failed

[dongahn]

Using the PMI_LIBRARY method led to the same problem (after environment variable cleanup):

sierra4367{dahn}74: env PMI_LIBRARY=/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0 FLUX_PMI_DEBUG=1 jsrun -a 1 -c 40 -n2 /usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/default/bin/flux start -o,-S,log-filename=out
PMI_Init: PMI_LIBRARY is set, use /g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
pmi_wrap_create: using /g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    no-plugins
from the file:
    help-pmix-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
-1: PMI_Init rc=1 PMI not initialized
flux-broker: PMI_Init: PMI not initialized
flux-broker: bootstrap failed
PMI_Init: PMI_LIBRARY is set, use /g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
pmi_wrap_create: using /g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    no-plugins
from the file:
    help-pmix-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
-1: PMI_Init rc=1 PMI not initialized
flux-broker: PMI_Init: PMI not initialized
flux-broker: bootstrap failed

[dongahn]

@rhc54: Any idea?

[trws]

That looks like libpmi failing to find the OpenMPI plugins it depends on. Did you build it to build in the paths or does it perhaps need OPAL_PATH or similar set to find everything?

On 27 Jun 2018, at 10:58, Dong H. Ahn wrote:

Using the PMI_LIBRARY method led to the same problem (after environment variable cleanup):

sierra4367{dahn}74: env 
PMI_LIBRARY=/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0 
FLUX_PMI_DEBUG=1 jsrun -a 1 -c 40 -n2 
/usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/default/bin/flux 
start -o,-S,log-filename=out
PMI_Init: PMI_LIBRARY is set, use 
/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
pmi_wrap_create: using 
/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    no-plugins
from the file:
    help-pmix-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
-1: PMI_Init rc=1 PMI not initialized
flux-broker: PMI_Init: PMI not initialized
flux-broker: bootstrap failed
PMI_Init: PMI_LIBRARY is set, use 
/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
pmi_wrap_create: using 
/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    no-plugins
from the file:
    help-pmix-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
-1: PMI_Init rc=1 PMI not initialized
flux-broker: PMI_Init: PMI not initialized
flux-broker: bootstrap failed

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/flux-framework/flux-core/issues/1555#issuecomment-400774946

[dongahn]

Through modules, our system is already setting $MPI_ROOT (=> /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-2018.06.01). Maybe there are other environment variables that the vanilla PMIX expects…

From: Tom Scogland notifications@github.com Reply-To: flux-framework/flux-core reply@reply.github.com Date: Wednesday, June 27, 2018 at 11:01 AM To: flux-framework/flux-core flux-core@noreply.github.com Cc: Dong Ahn ahn1@llnl.gov, Author author@noreply.github.com Subject: Re: [flux-framework/flux-core] jsrun on IBM Sierra doesn't bootstrap flux instance (#1555)

That looks like libpmi failing to find the OpenMPI plugins it depends on. Did you build it to build in the paths or does it perhaps need OPAL_PATH or similar set to find everything?

On 27 Jun 2018, at 10:58, Dong H. Ahn wrote:

Using the PMI_LIBRARY method led to the same problem (after environment variable cleanup):

sierra4367{dahn}74: env
PMI_LIBRARY=/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
FLUX_PMI_DEBUG=1 jsrun -a 1 -c 40 -n2
/usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/default/bin/flux
start -o,-S,log-filename=out
PMI_Init: PMI_LIBRARY is set, use
/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
pmi_wrap_create: using
/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
--------------------------------------------------------------------------
Sorry! You were supposed to get help about:
no-plugins
from the file:
help-pmix-runtime.txt
But I couldn't find that topic in the file. Sorry!
--------------------------------------------------------------------------
-1: PMI_Init rc=1 PMI not initialized
flux-broker: PMI_Init: PMI not initialized
flux-broker: bootstrap failed
PMI_Init: PMI_LIBRARY is set, use
/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
pmi_wrap_create: using
/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so.1.0.0
--------------------------------------------------------------------------
Sorry! You were supposed to get help about:
no-plugins
from the file:
help-pmix-runtime.txt
But I couldn't find that topic in the file. Sorry!
--------------------------------------------------------------------------
-1: PMI_Init rc=1 PMI not initialized
flux-broker: PMI_Init: PMI not initialized
flux-broker: bootstrap failed

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/flux-framework/flux-core/issues/1555#issuecomment-400774946

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/flux-framework/flux-core/issues/1555#issuecomment-400775748, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AA0nqx3cUIPr8lFdCDgT107uZQfyNkyHks5uA8hQgaJpZM4U4hl8.

[garlick]

In the pmix-2.1.1 sources, src/runtime/help-pmix-runtime.txt contains

[no-plugins]
We were unable to find any usable plugins for the %s framework. This PMIx
framework requires at least one plugin in order to operate. This can be caused
by any of the following:

* we were unable to build any of the plugins due to some combination
  of configure directives and available system support

* no plugin was selected due to some combination of MCA parameter
  directives versus built plugins (i.e., you excluded all the plugins
  that were built and/or could execute)

* the PMIX_INSTALL_PREFIX environment variable, or the MCA parameter
  "mca_base_component_path", is set and doesn't point to any location
  that includes at least one usable plugin for this framework.

[garlick]

Maybe pmix will need to be side installed, and PMIX_INSTALL_PREFIX set to point there?

[dongahn]

sierra4370{dahn}22: env PMI_LIBRARY=/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so PMIX_INSTALL_PREFIX=/g/g0/dahn/workspace/PMIX-PMI jsrun -a 1 -c 40 -n 2 /usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/default/bin/flux start
[sierra4370:59642] pmix_mca_base_component_repository_open: unable to open mca_ptl_tcp: /g/g0/dahn/workspace/PMIX-PMI/lib/pmix/mca_ptl_tcp.so: undefined symbol: pmix_bfrops_globals (ignored)
[sierra4370:59642] pmix_mca_base_component_repository_open: unable to open mca_ptl_usock: /g/g0/dahn/workspace/PMIX-PMI/lib/pmix/mca_ptl_usock.so: undefined symbol: pmix_gds_base_output (ignored)
flux-broker: PMI_Barrier: operation failed
flux-broker: bootstrap failed
flux-broker: PMI_Barrier: operation failed
flux-broker: bootstrap failed
E: (flux-broker) 18-06-27 11:33:13 dangling 'PAIR' socket created at src/zsys.c:379
E: (flux-broker) 18-06-27 11:33:13 dangling 'PAIR' socket created at src/zsys.c:380
E: (flux-broker) 18-06-27 11:33:13 dangling 'REP' socket created at src/zauth.c:77
E: (flux-broker) 18-06-27 11:33:13 dangling 'ROUTER' socket created at ../../../src/broker/overlay.c:472
E: (flux-broker) 18-06-27 11:33:13 dangling sockets: cannot terminate ZMQ safely
flux-broker: src/zsock_option.c:1347: zsock_set_sndtimeo: Assertion `rc == 0 || zmq_errno () == (156384712 + 53)' failed.
E: (flux-broker) 18-06-27 11:33:13 dangling 'PAIR' socket created at src/zsys.c:379
E: (flux-broker) 18-06-27 11:33:13 dangling 'PAIR' socket created at src/zsys.c:380
E: (flux-broker) 18-06-27 11:33:13 dangling 'REP' socket created at src/zauth.c:77
E: (flux-broker) 18-06-27 11:33:13 dangling 'ROUTER' socket created at ../../../src/broker/overlay.c:472
E: (flux-broker) 18-06-27 11:33:13 dangling sockets: cannot terminate ZMQ safely
flux-broker: src/zsock_option.c:1347: zsock_set_sndtimeo: Assertion `rc == 0 || zmq_errno () == (156384712 + 53)' failed.
ERROR:  One or more process terminated with signal 6 (core dumped)

Seems PMIX_INSTALL_PREFIX change the behavior and it looks to me like PMI_Init is satisfied. But this led to this problem.

[dongahn]

This is more informative. PMI_Barrier () is failing.

sierra4370{dahn}27: env FLUX_PMI_DEBUG=1 PMI_LIBRARY=/g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so PMIX_INSTALL_PREFIX=/g/g0/dahn/workspace/PMIX-PMI jsrun -a 1 -c 40 -n 1 /usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/default/bin/flux start
[sierra4370:72582] pmix_mca_base_component_repository_open: unable to open mca_ptl_tcp: /g/g0/dahn/workspace/PMIX-PMI/lib/pmix/mca_ptl_tcp.so: undefined symbol: pmix_bfrops_globals (ignored)
[sierra4370:72582] pmix_mca_base_component_repository_open: unable to open mca_ptl_usock: /g/g0/dahn/workspace/PMIX-PMI/lib/pmix/mca_ptl_usock.so: undefined symbol: pmix_gds_base_output (ignored)
PMI_Init: PMI_LIBRARY is set, use /g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so
pmi_wrap_create: using /g/g0/dahn/workspace/PMIX-PMI/lib/libpmi.so
0: PMI_Init rc=0
0: PMI_Get_size rc=0
0: PMI_Get_rank rc=0
0: PMI_Get_appnum rc=0
0: PMI_KVS_Get_name_length_max rc=0
0: PMI_KVS_Get_my_name ("7") rc=0
0: PMI_KVS_Get_key_length_max rc=0
0: PMI_KVS_Get_value_length_max rc=0
0: PMI_KVS_Put ("7", "cmbd.0.uri", "tcp://192.168.75.197:49152") rc=0
0: PMI_KVS_Commit rc=0
0: PMI_Barrier rc=-1 operation failed
flux-broker: PMI_Barrier: operation failed
flux-broker: bootstrap failed
E: (flux-broker) 18-06-27 11:36:09 dangling 'PAIR' socket created at src/zsys.c:379
E: (flux-broker) 18-06-27 11:36:09 dangling 'PAIR' socket created at src/zsys.c:380
E: (flux-broker) 18-06-27 11:36:09 dangling 'REP' socket created at src/zauth.c:77
E: (flux-broker) 18-06-27 11:36:09 dangling 'ROUTER' socket created at ../../../src/broker/overlay.c:472
E: (flux-broker) 18-06-27 11:36:09 dangling sockets: cannot terminate ZMQ safely
flux-broker: src/zsock_option.c:1347: zsock_set_sndtimeo: Assertion `rc == 0 || zmq_errno () == (156384712 + 53)' failed.
ERROR:  One or more process terminated with signal 6 (core dumped)

[garlick]

Maybe LD_LIBRARY_PATH needs to be set to pmix's lib directory so that pmix_bfrops_globals et al can be found?

[dongahn]

sierra4370{dahn}60: echo $LD_LIBRARY_PATH
/g/g0/dahn/workspace/PMIX-PMI/lib:/g/g0/dahn/workspace/PMIX-PMI/lib/pmix:/opt/ibm/spectrumcomputing/lsf/10.1/linux3.10-glibc2.17-ppc64le-csm/lib:/usr/tce/packages/cuda/cuda-9.2.88/lib64

Doesn't look like this make those unresolved symbols go away. And it is not clear if that's responsible for PMI_Barrier failure either.

[dongahn]

So, I am using libpmi.so which uses PMIx code from the same side installed PMIx. I am wondering the barrier problem is happening when this side installed PMIx is trying to talk to the PMIx server that jsrun uses to launch flux...

[dongahn]

FWIW, the PMIX variables jsrun sets for the job is the following:

PMIX_INSTALL_PREFIX=/usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-2018.06.01
PMIX_SYSTEM_TMPDIR=/var/tmp/jsm.sierra4370.7818/53640
PMIX_SERVER_TMPDIR=/var/tmp/jsm.sierra4370.7818/53640
PMIX_NAMESPACE=14
PMIX_RANK=0
PMIX_SERVER_URI2=pmix-server.122645;tcp4://127.0.0.1:50761
PMIX_SECURITY_MODE=native,none
PMIX_PTL_MODULE=tcp
PMIX_DSTORE_ESH_BASE_PATH=/var/tmp/jsm.sierra2991.7818/53640/pmix_dstor_122645

[garlick]

I am wondering the barrier problem is happening when this side installed PMIx is trying to talk to the PMIx server that jsrun uses to launch flux...

That's what I was assuming. Maybe PMI_Init suceeds without contacting the server, and the connect is attempted at the PMI_Barrier call. Then it fails because it fails to load those mca components "tcp" and "usock" due to unresolved symbols.

There is an option for building those plugins statically,

 --enable-mca-static=LIST
                          Comma-separated list of types and/or type-component
                          pairs that will be built statically linked into the
                          library. The default (if DSOs are supported) is to
                          build all components as DSOs. Enabling a component
                          as static disables it building as a DSO.

[dongahn]

pmix_bfrops_globals and pmix_gds_base_output are defined in the side installed libpmix.so

sierra4370{dahn}32: nm /g/g0/dahn/workspace/PMIX-PMI/lib/libpmix.so | grep pmix_bfrops_globals
0000000000114e98 B pmix_bfrops_globals
sierra4370{dahn}33:
sierra4370{dahn}33: nm /g/g0/dahn/workspace/PMIX-PMI/lib/libpmix.so | grep pmix_gds_base_output
0000000000111f10 D pmix_gds_base_output

Somehow, setting LD_LIBRARY_PATH to /g/g0/dahn/workspace/PMIX-PMI/lib didn't resolve this.

And when I actually preloaded it,

setenv LD_PRELOAD /g/g0/dahn/workspace/PMIX-PMI/lib/libpmix.so

this led to a JSM daemon failures.

sierra4370{dahn}38: setenv LD_PRELOAD /g/g0/dahn/workspace/PMIX-PMI/lib/libpmix.so
sierra4370{dahn}39: env PMIX_INSTALL_PREFIX=/g/g0/dahn/workspace/PMIX-PMI jsrun -a 1 -c 40 -n 1 hostname
ERROR: The connection to the JSM daemon has failed.
  This is a strong indication that the JSM daemon has exited unexpectedly.
  When this occurs, error logs are left on the affected nodes in
/tmp/JSMDIAG-LOG-<hostname>-<pid> or in
/var/tmp/jsm.sierra4370.7818/53647/JSMD-LOG-<hostname>-<pid>
READ:: Success
Failed to read status, RM_PMIx_spawn(): 0
ERROR: Other error.
[warn] event_add: event has no event_base set.

But they are not defined in the vendor-provided libpmix.so.

[dongahn]

At this point, I am not convinced that keep going down on this path will be profitable unless I hear back from @rhc54. Maybe, I should spend time instead to build a thin libpmi.so against the vendor installed libpmix.so

[garlick]

You might try grabbing the pmi1.c and pmi.h source files from the pmix source since that's pretty much what they are. I was able to get a clean build against the spectrum pmix public interfaces with only the following changes:

diff -u ../pmix-2.0.3/src/client/pmi1.c pmi1.c
--- ../pmix-2.0.3/src/client/pmi1.c 2017-10-16 08:30:06.000000000 -0700
+++ pmi1.c  2018-06-27 13:42:19.357667000 -0700
@@ -12,15 +12,9 @@
  * $HEADER$
  */

-#include <src/include/pmix_config.h>
-
-#include <src/include/types.h>
-
 #include <pmix.h>
 #include <pmi.h>

-#include "src/include/pmix_globals.h"
-
 #ifdef HAVE_STRING_H
 #include <string.h>
 #endif
@@ -30,14 +24,9 @@
 #ifdef HAVE_STDLIB_H
 #include <stdlib.h>
 #endif
-#include PMIX_EVENT_HEADER

 #define ANL_MAPPING "PMI_process_mapping"

-#include "src/buffer_ops/buffer_ops.h"
-#include "src/util/argv.h"
-#include "src/util/error.h"
-#include "src/util/output.h"

 #define PMI_MAX_ID_LEN       PMIX_MAX_NSLEN  /* Maximim size of PMI process group ID */
 #define PMI_MAX_KEY_LEN      PMIX_MAX_KEYLEN /* Maximum size of a PMI key */
@@ -182,9 +171,6 @@
         return PMI_SUCCESS;
     }

-    pmix_output_verbose(2, pmix_globals.debug_output,
-            "PMI_KVS_Put: KVS=%s, key=%s value=%s", kvsname, key, value);
-
     val.type = PMIX_STRING;
     val.data.string = (char*)value;
     rc = PMIx_Put(PMIX_GLOBAL, key, &val);
@@ -205,9 +191,6 @@
         return PMI_SUCCESS;
     }

-    pmix_output_verbose(2, pmix_globals.debug_output, "PMI_KVS_Commit: KVS=%s",
-            kvsname);
-
     rc = PMIx_Commit();
     return convert_err(rc);
 }
@@ -230,9 +213,6 @@
         return PMI_ERR_INVALID_VAL;
     }

-    pmix_output_verbose(2, pmix_globals.debug_output,
-            "PMI_KVS_Get: KVS=%s, key=%s value=%s", kvsname, key, value);
-
     /* PMI-1 expects resource manager to set
      * process mapping in ANL notation. */
     if (!strcmp(key, ANL_MAPPING)) {

[rhc54]

Sorry guys - was out for most of the day, and I have a few things I have to attend to before coming back online. I'll try to respond this evening. I can try to debug this using PRRTE as my host environment and see if I can identify the problem.

[dongahn]

@garlick: good idea.

[dongahn]

OK. I extracted pmi1.c as well as src/util/argv.[h|c] from PMIX and now can cleanly build libpmi.so against Spectrum installation. I can now bootstrap my own little test case as well as flux at 2 compute nodes! I will more testing.

For future reference, I will put my changes below (a bit more involved than @garlick's above.)

[dongahn]

sierra4368{dahn}47: diff -u /g/g0/dahn/workspace/PMIX-PMI/pmix-2.1.1/src/client/pmi1.c pmi1.c
--- /g/g0/dahn/workspace/PMIX-PMI/pmix-2.1.1/src/client/pmi1.c  2017-10-16 08:53:00.000000000 -0700
+++ pmi1.c  2018-06-27 14:23:12.008658000 -0700
@@ -12,33 +12,17 @@
  * $HEADER$
  */

-#include <src/include/pmix_config.h>
-
-#include <src/include/types.h>
-
 #include <pmix.h>
-#include <pmi.h>
+#include "pmi.h"
+#include "argv.h"

-#include "src/include/pmix_globals.h"
-
-#ifdef HAVE_STRING_H
+#include <stdio.h>
 #include <string.h>
-#endif
-#ifdef HAVE_UNISTD_H
 #include <unistd.h>
-#endif
-#ifdef HAVE_STDLIB_H
 #include <stdlib.h>
-#endif
-#include PMIX_EVENT_HEADER

 #define ANL_MAPPING "PMI_process_mapping"

-#include "src/mca/bfrops/bfrops.h"
-#include "src/util/argv.h"
-#include "src/util/error.h"
-#include "src/util/output.h"
-
 #define PMI_MAX_ID_LEN       PMIX_MAX_NSLEN  /* Maximim size of PMI process group ID */
 #define PMI_MAX_KEY_LEN      PMIX_MAX_KEYLEN /* Maximum size of a PMI key */
 #define PMI_MAX_KVSNAME_LEN  PMIX_MAX_NSLEN  /* Maximum size of KVS name */
@@ -182,9 +166,6 @@
         return PMI_SUCCESS;
     }

-    pmix_output_verbose(2, pmix_globals.debug_output,
-            "PMI_KVS_Put: KVS=%s, key=%s value=%s", kvsname, key, value);
-
     val.type = PMIX_STRING;
     val.data.string = (char*)value;
     rc = PMIx_Put(PMIX_GLOBAL, key, &val);
@@ -205,9 +186,6 @@
         return PMI_SUCCESS;
     }

-    pmix_output_verbose(2, pmix_globals.debug_output, "PMI_KVS_Commit: KVS=%s",
-            kvsname);
-
     rc = PMIx_Commit();
     return convert_err(rc);
 }
@@ -230,9 +208,6 @@
         return PMI_ERR_INVALID_VAL;
     }

-    pmix_output_verbose(2, pmix_globals.debug_output,
-            "PMI_KVS_Get: KVS=%s, key=%s value=%s", kvsname, key, value);
-
     /* PMI-1 expects resource manager to set
      * process mapping in ANL notation. */
     if (!strcmp(key, ANL_MAPPING)) {

sierra4368{dahn}48: diff -u /g/g0/dahn/workspace/PMIX-PMI/pmix-2.1.1/src/util/argv.h argv.h
--- /g/g0/dahn/workspace/PMIX-PMI/pmix-2.1.1/src/util/argv.h    2017-10-16 08:53:00.000000000 -0700
+++ argv.h  2018-06-27 14:21:40.093969000 -0700
@@ -34,7 +34,6 @@
 #ifndef PMIX_ARGV_H
 #define PMIX_ARGV_H

-#include <src/include/pmix_config.h>

 #ifdef HAVE_SYS_TYPES_H
@@ -43,8 +42,6 @@

 #include <pmix_common.h>

-BEGIN_C_DECLS
-
   /**
    * Append a string (by value) to an new or existing NULL-terminated
    * argv array.
@@ -74,7 +71,7 @@
    * value into the argv array; there is no need to keep the original
    * string (i.e., the arg parameter) after invoking this function.
    */
-PMIX_EXPORT pmix_status_t pmix_argv_append(int *argc, char ***argv, const char *arg) __pmix_attribute_nonnull__(1) __pmix_attribute_nonnull__(3);
+PMIX_EXPORT pmix_status_t pmix_argv_append(int *argc, char ***argv, const char *arg);

   /**
    * Append to an argv-style array, but ignore the size of the array.
@@ -151,7 +148,7 @@
    * argument (i.e., it can be freed after calling this function
    * without invalidating the output argv).
    */
-PMIX_EXPORT char **pmix_argv_split(const char *src_string, int delimiter) __pmix_attribute_malloc__ __pmix_attribute_warn_unused_result__;
+PMIX_EXPORT char **pmix_argv_split(const char *src_string, int delimiter);

   /**
    * Split a string into a NULL-terminated argv array. Include empty
@@ -168,7 +165,7 @@
    * argument (i.e., it can be freed after calling this function
    * without invalidating the output argv).
    */
-PMIX_EXPORT char **pmix_argv_split_with_empty(const char *src_string, int delimiter) __pmix_attribute_malloc__ __pmix_attribute_warn_unused_result__;
+PMIX_EXPORT char **pmix_argv_split_with_empty(const char *src_string, int delimiter);

   /**
    * Return the length of a NULL-terminated argv array.
@@ -198,9 +195,9 @@
    *
    * It is the callers responsibility to free the returned string.
    */
-PMIX_EXPORT char *pmix_argv_join(char **argv, int delimiter) __pmix_attribute_malloc__ __pmix_attribute_warn_unused_result__;
+PMIX_EXPORT char *pmix_argv_join(char **argv, int delimiter);

-PMIX_EXPORT char *pmix_argv_join_range(char **argv, size_t start, size_t end, int delimiter) __pmix_attribute_malloc__ __pmix_attribute_warn_unused_result__;
+PMIX_EXPORT char *pmix_argv_join_range(char **argv, size_t start, size_t end, int delimiter);

   /**
    * Return the number of bytes consumed by an argv array.
@@ -225,7 +222,7 @@
    * Specifically, the output argv will be an array of the same length
    * as the input argv, and strcmp(argv_in[i], argv_out[i]) will be 0.
    */
-PMIX_EXPORT char **pmix_argv_copy(char **argv) __pmix_attribute_malloc__ __pmix_attribute_warn_unused_result__;
+PMIX_EXPORT char **pmix_argv_copy(char **argv);

     /**
      * Delete one or more tokens from the middle of an argv.
@@ -301,6 +298,4 @@
  */
 PMIX_EXPORT pmix_status_t pmix_argv_insert_element(char ***target, int location, char *source);

-END_C_DECLS
-
 #endif /* PMIX_ARGV_H */

sierra4368{dahn}49: diff -u /g/g0/dahn/workspace/PMIX-PMI/pmix-2.1.1/src/util/argv.c argv.c
--- /g/g0/dahn/workspace/PMIX-PMI/pmix-2.1.1/src/util/argv.c    2017-10-16 08:53:00.000000000 -0700
+++ argv.c  2018-06-27 14:24:01.715941000 -0700
@@ -22,17 +22,9 @@
  * $HEADER$
  */

-#include <src/include/pmix_config.h>
-
-
-#ifdef HAVE_STDLIB_H
 #include <stdlib.h>
-#endif  /* HAVE_STDLIB_H */
-#ifdef HAVE_STRING_H
 #include <string.h>
-#endif  /* HAVE_STRING_H */
-
-#include "src/util/argv.h"
+#include "argv.h"

 #define ARGSIZE 128

CC     := xlc
CFLAGS := -g -O0 -fPIC
SPECTRUM_INSTALL := /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-2018.06.01

all: libpmi.so pmi_test

libpmi.so: pmi1.o argv.o
    $(CC) -shared $(CFLAGS) $^ -o $@ -L$(SPECTRUM_INSTALL)/lib -lpmix

pmi_test: pmi_test.o
    $(CC) $(CFLAGS) $^ -o $@ -L./ -lpmi -L$(SPECTRUM_INSTALL)/lib -lpmix

.c.o:
    $(CC) -I$(SPECTRUM_INSTALL)/include $(CFLAGS) -D_GNU_SOURCE=1 $^ -c -o $@

.PHONY:

clean: .PHONY
    rm -f *.o *~ *breakpoints*

[garlick]

I can now bootstrap my own little test case as well as flux at 2 compute nodes!

Nice one @dongahn!

[dongahn]

Thanks @garlick for your support! Will need to do some larger scale tests though.

[rhc54]

@garlick Yes, that is the "backing" code behind the PMI-1 APIs. The output_verbose functions will only work inside of PMIx. I'll try to see if our PMI-1 test codes work when running against a PMIx server, just to see if something is broken in the support.

[rhc54]

@dongahn I missed your updates - hooray! It looks from your changes that you mostly had to remove things that are defined in pmix_config.h after you removed that include. Correct? Just wondering what might have not been working before that is now fixed in case we have something that should go upstream.

[dongahn]

@rhc54:

Yes. In the morning, I built libpmi.so in tree and that didn't get me too far because of what appears to be a version mismatch of some sort between Spectrum's PMIX and the side-built PMIX.

So I carved out three files from the PMIX source tree and built the library against spectrum's public PMIX. That seems to work fine at small scale. I am testing flux with this method at larger scale to make sure jsrun can reliably bootstrap flux using this pmi-pmix combat library.

I think what might be useful would be to have a way to build libpmi.so against a public PMIX. As is, it depends on some internal PMIX stuff (things from pmix_config.h and argv stuff.)

As vendors adopt PMIX and may strip way its backward compatible pmi library, having an easy way to build libpmi seems useful.

Thank you for you help!

[rhc54]

I see - I can make that happen easily enough. I suspect Spectrum is using some older, probably v2.0.x, version. I'm meeting with them tomorrow and will ask. Glad you got over the hurdle!

[garlick]

We should ask IBM to distribute spectrum with the compat support enabled also.

[dongahn]

@We should ask IBM to distribute spectrum with the compat support enabled also.

Yes, I plan to do that. For now, I installed libpmi.so in /usr/global/tools/pmi4pmix/blueos_3_ppc64le_ib/lib/libpmi.so in case other folks want to try it.

[dongahn]

OK. I did some smoke tests at larger scales: 4, 64, 128, 512 and 1024 compute nodes.

sierra4362{dahn}52: bsub -nnodes $NNODES -Is -XF -G guests /usr/bin/tcsh
<<ssh X11 forwarding job>>
<<Waiting for dispatch ...>>
<<Starting on sierra4368>>

sierra4368{dahn}21: env PMI_LIBRARY=/usr/global/tools/pmi4pmix/blueos_3_ppc64le_ib/lib/libpmi.so jsrun -a 1 -c 40 -n ${NNODES} /usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/default/bin/flux start ip.sh -o,-S,log-filename=out

ip.sh initial program is:

#!/bin/sh

rm -f flux.info
hostname > flux.info
echo $FLUX_URI >> flux.info
sleep 3600

So when the initial program writes out the hostname and the URI into flux.info, I used that to connect my flux-proxy (within one of the front end nodes) to the broker rank in which the initial program is running:

/usr/global/tools/flux.sierra/blueos_3_ppc64le_ib/default/bin/flux proxy ssh://sierra2200//var/tmp/flux-kqxDb6

Then, I did a bunch of flux wreckrun|submit -N $N -n $N /usr/bin/hostname tests.

In this test, Flux worked reasonably okay. Two things I noticed though.

At 512 CNs, trying to exit out of the jsrun by typing in Ctrl-C led to an unclean exit. The jsrun appeared to hang and after giving some time, I had to kill -9 on the jsrun process.

But given that I didn't see this issue at 1024 CNs, I have to think this was an transient error in Sierra.

In addition, at larger scales like 512 and 1024 nodes, launching hostname can incur pretty large overhead. At 1024, it took on the order of 1 mins to complete this command. This might be a known issue though.

Closing this issue.