(Question) Where I can find the PMI1 wire protocol used by flux?

[rountree]

I'm at the point where I have successfully intercepted all of the Spectrum MPI PMIx calls and decoded them (including those that involve callbacks). I'll be working on translating those to PMI-1, and the step after that is getting the PMI-1 calls into Flux. I expect I'll be writing to the PMI_FD file descriptor in the environment. How do I format those call?

[rountree]

Dong suggests an alternative approach: add my LD_PRELOAD interceptors into the Flux PMI client library and let the existing client code handle talking to the Flux PMI server. The question then becomes how to call the client code from my interceptor code. Is this a more viable approach?

[garlick]

If you wanted to speak the wire protocol directly on PMI_FD, it is documented in Flux RFC 13. There is some code in flux-core/src/common/libpmi that could be borrowed if helpful: see simple_client.c, keyval.c, ... but the wire protocol is pretty simple and you may prefer to deal with it directly.

You could also dlopen the flux libpmi.so library and call PMI_Init() etc directly. Our PMI implementation doesn't use flux magic to get PMI service - it implements the client end of the standard wire protocol. I'm not completely sure this is what @dongahn was suggesting?

As I recall spectrum presents a challenge because a rank may have concurrent openmpi and pami threads calling into PMIx within a single rank. That's a challenge for either solution, although doing the wire protocol directly might make it easier to reason about.

[dongahn]

I'm not completely sure this is what @dongahn was suggesting?

Yes this is more or less I was suggesting (I think you can also directly link in the flux's PMI library into Barry's LD_PRELOAD module?)

I was hoping this way he doensn't have to worry about the wire protocol and let it be handled by Flux's client library.

The other question I had was on PMI2. Is PMI2 threadsafe? If PMI2 is implemented, can Barry's module use PMI2 without having to worry about the concurrency issue?

If this is the case, maybe what we should do is to implement PMI2 and have Barry's module to either directly build on it or use the same primitives used within the PMI2?

[garlick]

We don't implement PMI2 per se. There is a wrapper library that just maps some PMI2 functions to PMI1 counterparts that it turns out is pretty useless (see #2082)

See also:

1620 [spectrum mpi] PAMI requires PMIx service

1789 libpmi: support multiple PMI sessions per rank

1790 libpmi: support pre-populating KVS state

[dongahn]

@rountree: @garlick and I talked about this a bit after the meeting. It seems the path to least resistance would be to use the PMI1 wire protocol and resolve the concurrency issues locally at your LD_PRELOAD module.

[garlick]

Answered.