search

flux-framework / flux-pmix

flux shell plugin to bootstrap openmpi v5+
GNU Lesser General Public License v3.0
2 stars 4 forks source link

flux pmix plugin is installed with rpath pointing to ompi installed libdir #63

Closed garlick closed 1 year ago

garlick commented 1 year ago

Problem: when building flux-pmix with a side installed version of ompi, the shell plugin is installed with an rpath pointing to ompi's libpmix.so:

$ readelf -d /usr/local/lib/flux/shell/plugins/pmix/pmix.so |grep runpath
 0x000000000000001d (RUNPATH)            Library runpath: [/opt/ompi-v5.0.0rc9/lib]

This is despite having another version of openpmix installed in a default system location.

garlick commented 1 year ago

Generated src/shell/plugins/Makefile contains:

PMIX_LIBS = -L/opt/ompi-v5.0.0rc9/lib -Wl,-rpath -Wl,/opt/ompi-v5.0.0rc9/lib -Wl,--enable-new-dtags -lpmix

This is likely because we find both pmix and ompi using pkg-config


PKG_CHECK_MODULES([PMIX], [pmix >= 4.0.0])
...
PKG_CHECK_MODULES([OMPI], [ompi >= 3.0.0], [
...
and have to set PKG_CONFIG_PATH to the ompi directory in order to find it.
garlick commented 1 year ago

Openmpi is only needed here for testing. And we've pulled in stuff like the osu mpi benchmarks just so we can test openmpi with this plugin. Posilby that was as misstep.

Idea: drop the openmpi testing here and create a new mpi-test flux repo whose CI could test flux + flux-pmix + target MPI, where the target MPI is varied in CI.

garlick commented 1 year ago

As of #73, flux-pmix does not require openmpi to build, so this is avoidable by building without openmpi present, or by installing openpmix to a non side-installed path as we do in CI.