flux-framework / flux-core

core services for the Flux resource management framework
GNU Lesser General Public License v3.0
167 stars 49 forks source link

PMI2 Support Required for UnifyFS #2927

Open SteVwonder opened 4 years ago

SteVwonder commented 4 years ago

Per a conversation with the Unify team (@kathrynmohror, @adammoody, @MichaelBrim, @CamStan, @gonsie), we will need to expand our PMI2 support before we can bootstrap UnifyFS.

PMI2 interfaces used by Unify that we do not support currently:

PMI2_Job_GetRank
PMI2_Info_GetJobAttrIntArray
- "universeSize" (The size of the "universe" (defined for the MPI attribute MPI_UNIVERSE_SIZE))
- "hasNameServ" (The value hasNameServ is true if the PMI environment supports the name service operations (publish, lookup, and unpublish))

PMI2 interfaces used by Unify that we do support:

PMI2_Initialized
PMI2_Init
PMI2_Job_GetId
PMI2_Finalize
PMI2_KVS_Get*
PMI2_KVS_Put
PMI2_KVS_Fence

* Need to verify that a "NULL" kvsname works

@garlick would know best, but I think implementing just those three things should be relatively easy and straightforward. We would return true for hasNameServ, and then lookup rank and size info (that we certainly already know about) for universeSize and Job_GetRank. At minimum, we can get that info by reading FLUX_JOB_SIZE and FLUX_TASK_RANK from the environment, but I assume there are cleaner ways involving the Flux KVS.

Reference to UnifyFS's PMI2 usage: https://github.com/LLNL/UnifyFS/blob/dev/common/src/unifyfs_keyval.c

garlick commented 4 years ago

I think you're right that this is easy, at least on the surface of it. Couple things we should check out: