RPATH with Python and virtualenv

[rjeschmi]

One use case I'd like to see work is to have a version of Python that works properly with virtualenv without

after compiling Python with --rpath I can see that RPATH is set with readelf:

0x000000000000000f (RPATH) Library rpath: [$ORIGIN/../lib:$ORIGIN/../lib64:/software/software/Core/GCCcore/4.9.3/lib64:/software/software/Core/GCCcore/4.9.3/lib:/software/software/Compiler/GCC/4.9.3-2.25/OpenBLAS/0.2.15-LAPACK-3.6.0/lib:/software/software/MPI/GCC/4.9.3-2.25/OpenMPI/1.10.2/ScaLAPACK/2.0.2-OpenBLAS-0.2.15-LAPACK-3.6.0/lib:/software/software/MPI/GCC/4.9.3-2.25/OpenMPI/1.10.2/FFTW/3.3.4/lib:/software/software/MPI/GCC/4.9.3-2.25/OpenMPI/1.10.2/bzip2/1.0.6/lib:/software/software/Core/zlib/1.2.8/lib:/software/software/MPI/GCC/4.9.3-2.25/OpenMPI/1.10.2/libreadline/6.3/lib:/software/software/MPI/GCC/4.9.3-2.25/OpenMPI/1.10.2/ncurses/6.0/lib:/software/software/MPI/GCC/4.9.3-2.25/OpenMPI/1.10.2/SQLite/3.9.2/lib:/software/software/MPI/GCC/4.9.3-2.25/OpenMPI/1.10.2/Tk/8.6.4-no-X11/lib:/software/software/MPI/GCC/4.9.3-2.25/OpenMPI/1.10.2/GMP/6.1.0/lib:/software/software/Core/GCCcore/4.9.3/lib64/../lib64:/software/software/Core/GCCcore/4.9.3/lib/../lib64:/software/software/Core/GCCcore/4.9.3/lib/gcc/x86_64-unknown-linux-gnu/4.9.3:/software/software/Core/GCCcore/4.9.3/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/../../../../lib64:/software/software/Core/binutils/2.25/lib:/software/software/MPI/GCC/4.9.3-2.25/OpenMPI/1.10.2/Tcl/8.6.4/lib:/software/software/Compiler/GCC/4.9.3-2.25/OpenMPI/1.10.2/lib:/software/software/Compiler/GCC/4.9.3-2.25/hwloc/1.11.2/lib:/software/software/Compiler/GCC/4.9.3-2.25/numactl/2.0.11/lib:/software/software/Core/GCCcore/4.9.3/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/../../..]

[rob@ottbioinfo venv]$ ./py-eb/bin/python --version
Python 2.7.5
[rob@ottbioinfo venv]$ ldd py-eb/bin/python
        linux-vdso.so.1 =>  (0x00007ffc017b3000)
        libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007efdd4b32000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007efdd492d000)
        libm.so.6 => /lib64/libm.so.6 (0x00007efdd462b000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007efdd440f000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007efdd420b000)
        libc.so.6 => /lib64/libc.so.6 (0x00007efdd3e49000)
        /lib64/ld-linux-x86-64.so.2 (0x00007efdd4f02000)

[boegel]

@rjeschmi What kind of specific errors are you running into with virtualenv when a Python that includes an RPATH on $ORIGIN/../lib and $ORIGIN/../lib64?

I think the only way around this is to avoid using $ORIGIN, and just inject the actual installation prefix + /lib in there instead.

Interestingly, Spack is actually considering switching to $ORIGIN (rather than hardcoding the installation prefix) to make installations easier to relocate, see https://github.com/LLNL/spack/issues/640 .

[boegel]

@pescobar You should be aware of this... (virtualenv is broken when Python was installed with RPATH the way it is implemented now)

[pescobar]

@rjeschmi thanks for pointing this.

This is a blocker for me because my users do heavy usage of virtualenvs. I am thinking if I should totally stop using rpath or just use the rpath skip-out option to build python.

I guess a similar issue can happen if users install R packages to their home folder and R has been linked with rpath, isn't it?

[rjeschmi]

@pescobar after talking online with you I don't think it changes the behaviour for you vs regular easybuild right now.

I don't think it will effect any other packages either. It is just a relocation peculiarity and if you have LD_LIBRARY_PATH set, it will still work fine (just RPATH isn't doing the work)

[pescobar]

@rjeschmi thanks for the explanation in the irc channel ;)

As you mention, this issue would not affect my specific use case because I still define LD_LIBRARY_PATH in my modules but I agree that getting virtualenvs to work without needing any extra env var would be nice

[boegel]

@rjeschmi Do you see any reason to just stop using $ORIGIN and just hard-code the installation path instead?

Especially w.r.t. Spack considering to move to $ORIGIN...

[boegel]

Relevant to this discussion is @ocaisa's comment in https://github.com/hpcugent/easybuild/issues/282#issuecomment-262721484

[rjeschmi]

The only reason to use $ORIGIN is if you are going to support relocation. I don't mind it in theory, but it should be a further experimental option.

[ocaisa]

Isn't the installation step itself usually a relocation? What if the application requires loading of it's own .so libraries (like Qt does)? If installation is a relocation then you can only ensure they are loaded correctly by using $ORIGIN

[boegel]

So, that would mean building Qt with --rpath would be broken if we switch from $ORIGIN/../lib to RPATH'ing <prefix>/lib instead?

[ocaisa]

No it wouldn't most likely...but that's assuming that your choice of location is actually the correct one

[boegel]

@tgamblin Any thoughts on this on the benefits over RPATH'ing $ORIGIN/../lib vs <prefix>/lib? Especially w.r.t. https://github.com/LLNL/spack/issues/640

[ocaisa]

To me, one for the other is a straight swap so I say go with hard-coding as the default but allow people to switch to $ORIGIN if they want (or as needed).

[tgamblin]

I'd just use prefix here. If you look at the RPATH on the Python binary, it has tons of absolute paths to dependencies anyway, so $ORIGIN isn't buying you relocatability. You'd have to change all those other non-ORIGIN RPATHs too if you wanted to move your EB installation.

I am not sure we'll move to relative RPATHs soon because it's hard to get right. You have to know a lot about how your build is laid out -- e.g., how many levels deep in the prefix is the library? It looks like you assume one-level deep libs here ($prefix/lib[64]), but what if your binary is in a subdirectory of lib? Then you need to know to do $ORIGIN/../.. and not just $ORIGIN/... It is hard to know the right thing to do at build time because you don't actually know where the lib will be installed.

Because it's so hard to do this in a general way, Spack probably won't move to relative RPATHs any time soon. I am thinking we could allow relocation more easily with patchelf than with $ORIGIN. See LLNL/spack#445.

[boegel]

Thanks a lot for the feedback @tgamblin! :+1:

[rjeschmi]

Yeah you have the advantage too that $ORIGIN/../libs will probably fit in the same space as $prefix (but less likely the other way) so patchelf should be a lot easier that direction. Also the patchelf should be relatively easy.

Definitely worth thinking about long_fake_prefix that is easy to search for as an option with a patchelf finish step (and this is all easier if the prefix is just there in the RPATH).

[tgamblin]

I believe parchelf works fine with variable-length/longer RPATHs. It's older cousin chrpath doesn't.

[jdonners]

I agree with @tgamblin that using the absolute paths is better for libs that are installed in deeper hierarchies (I'm thinking especially shared objects for python packages here). However, I'd suggest to include $ORIGIN itself as well. E.g. ParaView installs libraries in lib/paraview-$version, which wouldn't be found otherwise.

[mzpqnxow]

I had success making this work by symlinking to lib/lib64 directories some time back, but this was done with a tool that wrapped virtualenv, not in virtualenv itself...

[morrisonlevi]

Drive-by comment: I think $ORIGIN ought to be supported but then again I'm just a passer-by with no additional info other than it seems odd that the "better practice" doesn't work. Perhaps it ought to be considered an upstream bug, though.

[boegel]

@pescobar Is this actually still a problem? Isn't this fixed with the change made in https://github.com/easybuilders/easybuild-framework/pull/2358 (which is part of EasyBuild v3.5.0 & newer)?