rpath incompatible with LD_LIBRARY_PATH export

[EricDeveaud]

Hello,

after installing goolf-1.7.20 toolchain using --rpath support I was surprised to notice that generated modulefiles for the tools in toolchain still export LD_LIBRARY_PATH.

eg:

bigmess:~ > module list    
No modules loaded

bigmess:~ > env | grep LD_LIBRARY_PATH
bigmess:~ > module load goolf/1.7.20  
bigmess:~ > env | grep LD_LIBRARY_PATH
LD_LIBRARY_PATH=/soft/exe/ScaLAPACK/2.0.2-gompi-1.7.20-OpenBLAS-0.2.13-LAPACK-3.5.0/lib:/soft/exe/FFTW/3.3.4-gompi-1.7.20/lib:/soft/exe/OpenBLAS/0.2.13-GCC-4.8.4-LAPACK-3.5.0/lib:/soft/exe/OpenMPI/1.8.4-GCC-4.8.4/lib:/soft/exe/hwloc/1.10.1-GCC-4.8.4/lib:/soft/exe/numactl/2.0.10-GCC-4.8.4/lib:/soft/exe/GCC/4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4:/soft/exe/GCC/4.8.4/lib64:/soft/exe/GCC/4.8.4/lib

In my opinionn --rpath should disable the LD_LIBRARY_PATH export

furthermore I think that in order to maintain a consistency across all the easybuild generated builds --rpath or not must be under a site configuration option that will prevent mixing rpath installation and non rpath ones.

in order to investigate setting rpath site config parameter to true I will wipe out everything and rebuild goolf-1.7.20 with this setting

regards

Eric

[ocaisa]

I don't quite agree with this, if someone wants to link to a library having LD_LIBRARY_PATH makes life easier for them. However I do agree that when I load a module I should only get the LD_LIBRARY_PATH for that module, there should be no load dependencies in an rpath-ed module (not from it's dependencies nor the toolchain).

The example you've used isn't the most helpful in this case because when you load a toolchain, like goolf, you are explicitly loading the components it is made up of so I would almost expect the output you got (but not hwloc or numactl since these should be rpath-ed in and not explicitly loaded)

[boegel]

There's a chicking-or-egg situation w.r.t. using --rpath and updating $LD_LIBRARY_PATH in the generated module file, see also http://easybuild.readthedocs.io/en/latest/RPATH-support.html#relation-to-ld-library-path, so I agree with @ocaisa here.

W.r.t. the dependencies: we'll need to distinguish runtime dependencies from (what I call) link-time dependencies, see also https://github.com/hpcugent/easybuild-framework/pull/1960 . Some dependencies only need to be available at build(/link) time when using --rpath, while they're still required at runtime without using --rpath. Others will still be runtime dependencies though, i.e. stuff that also provides binaries (e.g. Python). If whatever you're installing requires or calls out to binaries provided by one of the deps, that deps will need to be loaded at runtime too.

The point about toolchains is an interesting one, and goes back to a long-standing discussion, see https://github.com/hpcugent/easybuild-framework/issues/570 .

@EricDeveaud being consistent w.r.t. use of --rpath is up to each site, some may want to mix installations with/without RPATH. If you want that as a system-wide setting, you can enable the rpath configure option in a system-wide EasyBuild configuration file (cfr. http://easybuild.readthedocs.io/en/latest/Configuration.html#configuration-file-s)

[ocaisa]

I see your point, there certainly is a need to support the concept of link-time dependencies...but then certainly the toolchains are purely link-time dependencies and shouldn't appear as dependencies in the final module file. At the very least the appearance of link-time dependencies in the module files should be configurable.

[jbd]

"if someone wants to link to a library having LD_LIBRARY_PATH makes life easier for them."

I agree with the spirit of making life easier. But using LD_LIBRARY_PATH could have some side effects mentionned in https://github.com/hpcugent/easybuild/issues/274 for exemple. From what I understand from the problem, LD_RUN_PATH seems a better choice to make life easier for people while keeping them away from thoses side effects (or introducing worse ones :D).

Maybe I'm missing something here, but if LD_RUN_PATH is a reasonnable choice easybuild could offer the choice to alter LD_LIBRARY_PATH or LD_RUN_PATH (site-wide) when loading modules ?

[ocaisa]

I can't think of any real problem with that off the top of my head, might be an interesting option to support...but would play very badly with the current rpath implementation (from how I understand things) so you couldn't have both.

[ocaisa]

Hmm, had a quick look, logically it's a straightforward swap but practically it needs a lot of refactoring in the easyblocks repo and a lot of checking in the easyconfigs repo. I would've thought the initial instance would pop up in the framework but it seems it's really coming from the easyblocks.

Nice project for the User Meeting perhaps... @EricDeveaud @jbd are you interested in coming?

[ocaisa]

Or maybe just switch one out for the other in the module generator: https://github.com/hpcugent/easybuild-framework/blob/master/easybuild/tools/modules.py#L577

It's dirty...but quick

[boegel]

Replacing LD_LIBRARY_PATH with LD_RUN_PATH in module_generator:prepend_paths (not modules.py, which deals with loading/searching of modules, not generating module files) is certainly the least intrusive way of doing this.

I don't think the equivalent of adding logic to check whether LD_LIBRARY_PATH or LD_RUN_PATH should be set in various places (framework, easyblocks) makes a lot of sense w.r.t. maintainability, and that's still ignoring the fact that also easyconfig files may specify (additional) paths to be included in $LD_LIBRARY_PATH

This should probably be steered via a configure option like --use-ld-run-path...

Who's up for taking a stab at this?

[jbd]

Wow, lot of suggestions that worth having a look here. Thank you for the feedback ! We need to look into it while continuing our deep dive into Easybuild.

Coming to User Meeting would be very cool. How much time before the meeting date do you need the registration to be done ?

[boegel]

@jbd you'll be able to register until a couple of days before the user meeting, let's say the deadline is Feb 1st (unless we run out of seats by then); more details on registering very soon, we're working on that (cc @damianam)

[ocaisa]

The one issue I see with swapping LD_LIBRARY_PATH for LD_RUN_PATH is the case where an executable from a module is linked against some of it's own libraries, these libraries will get moved during the install step meaning the executable won't find them without LD_LIBRARY_PATH at run time. This is a problem which is solved with the rpath support by leveraging $ORIGIN. I can't see completely see how this can be fixed for LD_RUN_PATH...you can do something naive like redefine LD_RUN_PATH=\$ORIGIN/../lib:\$ORIGIN/../lib64:$LD_RUN_PATH as a prebuildopt to at least insert some sensible defaults for this case. You could also perhaps trigger the rpath sanity check (or a variant) to check the end results doesn't have any missing links? I'm not sure that covers every scenario though.

[EricDeveaud]

thanks for inviting to user meeting, will check if it's possible with my boss.

regarding using LD_RUN_PATH and moving libraries is not a problem, the only case I know where you have to take care of this situation is when upstream build chain does not handle the installation correctly. it is just a matter of setting some compil flags or some cmake options in order to correct this problem. as some *.eb.configopts easycinfig files already do.

[ocaisa]

Setting some compile flags or CMake options for 1000s of easyconfigs is not exactly a small task...

[EricDeveaud]

who said that software installation is an easy task ?

sorry could not resist

[damianam]

who said that software installation is an easy task ?

EasyBuild. I couldn't resist either ;-P