search

hpc / mpifileutils

File utilities designed for scalability and performance.
https://hpc.github.io/mpifileutils
BSD 3-Clause "New" or "Revised" License
164 stars 64 forks source link

How to disable rpaths? #395

Open brianjmurrell opened 3 years ago

brianjmurrell commented 3 years ago

I notice in the cmake output:

-- MPI C Link Flags:     -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags
...
-- MPI CXX Link Flags:    -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags

How can those rpaths be disabled? Those are standard paths and will be searched without rpaths. It's a problem in particular for RPM packaging because RPM packaging on RedHat distros forbids the use of rpaths and breaks the build if they are found.

adammoody commented 3 years ago

I'm trying out where it comes form to start with. I think the CMake script that fills in those variables is pulling the values from the mpicc and mpicxx scripts from the installed MPI library. Do you see those flags in the output of mpiccc -show?

brianjmurrell commented 3 years ago 
$ mpicc -show
gcc -std=gnu99 -std=gnu99 -m64 -O2 -fPIC -Wl,-z,noexecstack -I/usr/include/mpich-x86_64 -L/usr/lib64/mpich/lib -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags -lmpi
daltonbohning commented 3 years ago

Based on this: https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling

It seems that mpifileutils is intentionally turning rpath on: https://github.com/hpc/mpifileutils/blob/fb5b43cd703bdf4d2857238fb5969525e688242b/CMakeLists.txt#L133-L161

It looks like we could SET(CMAKE_SKIP_RPATH TRUE) to disable rpath completely: https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling#no-rpath-at-all

Maybe we can have a compile flag for doing this. Maybe something like:

IF(DISABLE_RPATH)
    SET(CMAKE_SKIP_RPATH TRUE)
ELSE
    # use above code
ENDIF

I'm investigating now to see the plausibility of this

daltonbohning commented 3 years ago

@brianjmurrell Will you try the branch rpath-test and run cmake with the option -DENABLE_RPATH=OFF and then do a make install? You will still see -rpath in the cmake output, but if you run readelf -d /dir/to/libmfu.so then -rpath should be gone.

After running this, is the rpath issue solved for the rpm packaging?

brianjmurrell commented 3 years ago

Interestingly, I am unable to reproduce this issue any more.

daltonbohning commented 3 years ago

It seems that cmake always shows -rpath when you configure, but when you make install, -rpath is only enabled for non-system directories: https://github.com/hpc/mpifileutils/blob/fb5b43cd703bdf4d2857238fb5969525e688242b/CMakeLists.txt#L153-L157

Either way, glad it's working. Let me know if a similar issue crops up again.

adammoody commented 3 years ago

@daltonbohning , I like the idea of having a -DENABLE_RPATH=OFF CMake option to selectively disable rpath. Having rpath is helpful for users who are installing to non-system paths, since that saves people the trouble of setting LD_LIBRARY_PATH before running a tool. Good idea.

@brianjmurrell , as for the rpath to MPI in particular, I think the CMake FindMPI script pulls its values from the output of mpicc -show, and it looks like mpicc is defining an rpath entry from your output above.

gcc -std=gnu99 -std=gnu99 -m64 -O2 -fPIC -Wl,-z,noexecstack -I/usr/include/mpich-x86_64 -L/usr/lib64/mpich/lib -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags -lmpi

I don't know whether CMake will detect and strip out rpath entries that exist from the mpicc -show output when someone tells CMake itself to avoid using rpaths. Is that what you're seeing? Does your mpicc output still show an rpath entry?

brianjmurrell commented 3 years ago

It seems it does:

$ mpicc --show
gcc -I/usr/include/openmpi3-x86_64 -pthread -Wl,-rpath -Wl,/usr/lib64/openmpi3/lib -Wl,--enable-new-dtags -L/usr/lib64/openmpi3/lib -lmpi

I used openmpi3 here as that one definitely comes from my distro and is therefore representative of the distro's packaging standards. That said, I am quite surprised about this as the Fedora packaging guidelines forbid the use of rpaths.

All of that said, https://github.com/hpc/mpifileutils/issues/395#issuecomment-702199449. Not sure why it was being an issue at the time.

daltonbohning commented 3 years ago

@adammoody Based on: https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling#always-full-rpath

It seems that our mpifileutils/CMakeLists.txt is identical to example shown for Always full RPATH, and that rpath is always used, unless the library is one of the default system libraries, which I believe /usr/... would be in Brian's case.

And since Brian is no longer having the issue, do you think we should still add the -DENABLE_RPATH=OFF` option? (I already have a branch for this, and it seems trivial enough)