search

TACC / Lmod

Lmod: An Environment Module System based on Lua, Reads TCL Modules, Supports a Software Hierarchy
http://lmod.readthedocs.org
Other
475 stars 122 forks source link

Incorrect LD_LIBRARY_PATH after load. #111

Closed wpoely86 closed 8 years ago

wpoely86 commented 8 years ago

Using 6.1.7:

$ echo $LD_LIBRARY_PATH

$ ml CP2K/2.6.0-intel-2015a
$ echo $LD_LIBRARY_PATH
/apps/gent/CO7/haswell-ib/software/libxc/2.2.1-intel-2015a/lib:/apps/gent/CO7/haswell-ib/software/Libint/1.1.4-intel-2015a/lib:/apps/gent/CO7/haswell-ib/software/imkl/11.2.1.133-iimpi-7.2.3-GCC-4.9.2/mkl/lib/intel64:/apps/gent/CO7/haswell-ib/software/imkl/11.2.1.133-iimpi-7.2.3-GCC-4.9.2/lib/intel64:/apps/gent/CO7/haswell-ib/software/impi/5.0.2.044-iccifort-2015.1.133-GCC-4.9.2/lib64:/apps/gent/CO7/haswell-ib/software/ifort/2015.1.133-GCC-4.9.2/lib/intel64:/apps/gent/CO7/haswell-ib/software/ifort/2015.1.133-GCC-4.9.2/lib:/apps/gent/CO7/haswell-ib/software/icc/2015.1.133-GCC-4.9.2/lib/intel64:/apps/gent/CO7/haswell-ib/software/icc/2015.1.133-GCC-4.9.2/lib:/apps/gent/CO7/haswell-ib/software/GCC/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2:/apps/gent/CO7/haswell-ib/software/GCC/4.9.2/lib64:/apps/gent/CO7/haswell-ib/software/GCC/4.9.2/lib
$ ml vsc-base/2.1.2
$ echo $LD_LIBRARY_PATH
/apps/gent/CO7/haswell-ib/software/vsc-base/2.1.2/lib

$ ml purge
The following modules were not unloaded:
   (Use "module --force purge" to unload all):

  1) cluster/swalot

$ ml CP2K/2.6.0-intel-2015a vsc-base/2.1.2
$ echo $LD_LIBRARY_PATH
/apps/gent/CO7/haswell-ib/software/vsc-base/2.1.2/lib:/apps/gent/CO7/haswell-ib/software/libxc/2.2.1-intel-2015a/lib:/apps/gent/CO7/haswell-ib/software/Libint/1.1.4-intel-2015a/lib:/apps/gent/CO7/haswell-ib/software/imkl/11.2.1.133-iimpi-7.2.3-GCC-4.9.2/mkl/lib/intel64:/apps/gent/CO7/haswell-ib/software/imkl/11.2.1.133-iimpi-7.2.3-GCC-4.9.2/lib/intel64:/apps/gent/CO7/haswell-ib/software/impi/5.0.2.044-iccifort-2015.1.133-GCC-4.9.2/lib64:/apps/gent/CO7/haswell-ib/software/ifort/2015.1.133-GCC-4.9.2/lib/intel64:/apps/gent/CO7/haswell-ib/software/ifort/2015.1.133-GCC-4.9.2/lib:/apps/gent/CO7/haswell-ib/software/icc/2015.1.133-GCC-4.9.2/lib/intel64:/apps/gent/CO7/haswell-ib/software/icc/2015.1.133-GCC-4.9.2/lib:/apps/gent/CO7/haswell-ib/software/GCC/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2:/apps/gent/CO7/haswell-ib/software/GCC/4.9.2/lib64:/apps/gent/CO7/haswell-ib/software/GCC/4.9.2/lib

Am I missing something here? The contents of the vsc-base/2.1.2 module is

#%Module
proc ModulesHelp { } {
    puts stderr { VSC-tools is a set of Python libraries and scripts that are commonly used within HPC-UGent. - Homepage: http://hpcugent.github.com/vsc-mympirun/
    }
}

module-whatis {Description: VSC-tools is a set of Python libraries and scripts that are commonly used within HPC-UGent. - Homepage: http://hpcugent.github.com/vsc-mympirun/}

set root /apps/gent/CO7/haswell-ib/software/vsc-base/2.1.2

conflict vsc-base

prepend-path    LD_LIBRARY_PATH     $root/lib
prepend-path    LIBRARY_PATH        $root/lib
prepend-path    PATH        $root/bin
setenv  EBROOTVSCMINBASE        "$root"
setenv  EBVERSIONVSCMINBASE     "2.1.2"
setenv  EBDEVELVSCMINBASE       "$root/easybuild/vsc-base-2.1.2-easybuild-devel"

prepend-path    PYTHONPATH      $root/lib
# Built with EasyBuild version 2.1.0dev
wpoely86 commented 8 years ago

ml -D -D output of both loading only vsc-base/2.1.2 and loading CP2K/2.6.0-intel-2015a vsc-base/2.1.2 can be found at: https://gist.github.com/wpoely86/f362f99cc233a0ab3571cd0e38db8455

boegel commented 8 years ago

wow, that's weird... and worrying

wpoely86 commented 8 years ago

Ignore, this is our internal mess.

boegel commented 8 years ago

Well, hold on, I think it's worth clarifying what's going on here...

Our definition of ml and module does this, in an attempt to avoid breaking lua because of changes to $LD_LIBRARY_PATH:

eval $(LD_LIBRARY_PATH="" $LMOD_CMD bash "$@");

However, that causes Lmod to see the wrong value of $LD_LIBRARY_PATH, breaking updates to that variable.

@rtmclay: thoughts?

wpoely86 commented 8 years ago

@rtmclay can Lmod restore the LD_LIBRARY_PATH on every load based on it's internal state? Or can we pass the correct value as an option to $LMOD_CMD?

rtmclay commented 8 years ago

My recommendation is to have Lmod a new hook. One that gets called at the start Lmod execution. Then a site like your can do what you want.

I don't have time to work on it now but I would take a pull request.

boegel commented 8 years ago

@rtmclay Do you mean like a 'startup hook', i.e. one that will be triggered every time Lmod runs (as opposed to other hooks that only get triggered on particular subcommands)?

Any pointers on how that could be implemented, which files/functions to look at, anything that comes close (load hook, something better)?

wpoely86 commented 8 years ago

I'm happy to have a look at it.

But just out of curiosity: can the library path be reconstructed from the 'saved' internal state?

rtmclay commented 8 years ago

Well, you could do something like:

module () { eval $( MY_LD_LIB_PATH=$LD_LIBRARY_PATH LD_LIBRARY_PATH="" $LMOD_CMD bash "$@");}

Then inside the startup_hook (to use Kenneth's idea for a name) set 

   LD_LIBRARY_PATH=$MY_LD_LIB_PATH

I'd look at the load hook or any other and call it just after the SitePackage is loaded in lmod.in.lua. That is what the lmod script is called in the source directory.

boegel commented 8 years ago

Actually, isn't the load hook good enough?

Only there is the value for $LD_LIBRARY_PATH relevant w.r.t. changing it, right?

Well, also unload & purge (same thing?), I guess.

wpoely86 commented 8 years ago

It's much cleaner to do it as early on as possible.

rtmclay commented 8 years ago

The load hook may be sufficient. It is certainly an easy thing to test. But I agree with @wpoely86 that it is much cleaner to do it early.