execute a bash script within the modulesfile

[amamory]

Is your feature request related to a problem? Please describe.

There tools that already have their own setup script, to set their environment variables. However, they lack the ability to unload them, as 'module' does.

Describe the solution you'd like

Module1.0#####################################################################
##
## modules Xilinx
##
# Local variables
set MODULE "xilinx"
set VERSION "2019.2"
set BASE "/home/lsa/xilinx/Xilinx"

# License 
setenv XILDIR /home/lsa/xilinx/Xilinx

# Helps
module-whatis "Xilinx and Petalinux environment"
# unfortunately, it is not possible to pass the environment variables to the calling term :(
#exec bash /home/lsa/xilinx/Xilinx/Vitis/2019.2/settings64.sh
#exec bash /home/lsa/xilinx/Xilinx/petalinux/settings.sh

#if { [catch { exec /bin/bash -c "source /home/lsa/xilinx/Xilinx/Vitis/2019.2/settings64.sh" } msg] } {
#   puts "ERROR runnig source /home/lsa/xilinx/Xilinx/Vitis/2019.2/settings64.sh"
#   puts "Information about it: $::errorInfo"
#}
#if { [catch { exec /bin/bash -c "source /home/lsa/xilinx/Xilinx/petalinux/settings.sh" } msg] } {
#   puts "ERROR runnig source /home/lsa/xilinx/Xilinx/petalinux/settings.sh"
#   puts "Information about it: $::errorInfo"
#}

# Simple user information
if { [ module-info mode load ] } {
        puts stderr "\n\ttype: source /home/lsa/xilinx/Xilinx/Vitis/2019.2/settings64.sh"
        puts stderr "\n\ttype: source /home/lsa/xilinx/Xilinx/petalinux/settings.sh"
}

if { [ module-info mode remove ] } {
        puts stderr "\n\topen a new terminal to revome the definitions\n"
}

The commented lines execute, but the environment variables are not set within 'modules'. However, if I run the same script directly in the bash terminal, it works. The variables are created.

Describe alternatives you've considered

These bash scripts are usually complex. They dont only set few env vars. So translating them to TCL is not an option.

Any idea to make this work ?

[xdelaruelle]

I am currently working on such feature. It will consist on a new modulefile command named source-sh:

source-sh shell script [arg...]

It will start designated shell to:

1. get current environment state (environment variables, shell aliases, shell functions and current working dir)
2. source designated script with args
3. get resulting environment state

Once done, environment prior and after script source are compared to determine the corresponding environment changes and translate those changes into modulefile commands (setenv, prepend-path, set-alias, set-function, ...).

Those commands are then evaluated as if there were written directly in the modulefile. Special care will be given to track those resulting modulefile commands, in order to be able to undo them when modulefile is unloaded.

[amamory]

Great! I'll be monitoring this new feature.

Thanks for the feedback.

[xdelaruelle]

source-sh modulefile command has been introduced (fdebca900558c2261d1726c0bb6d5f2424537db8), it will be include in next feature release (4.6)

[ernstki]

Hi everyone, hope you're well. Thanks Xavier, for all your efforts keeping the Modules project going.

Just for edification purposes, is something like this an acceptable workaround for users stuck with 3.2.x (because it's in the official repos for the OS on our cluster)?

puts source /home/lsa/xilinx/Xilinx/Vitis/2019.2/settings64.sh

Modulefiles, as I understand them, eval their (stdout) output in the context of the current shell environment, and I'm pretty sure I've used this technique with success in the past.

Two potential problems I foresee:

• if you attempt to source a script written for a different shell than the user's login shell (e.g. it's Bourne shell script, but the user is running the C shell)
• module unload cannot "un-source" that file, so any environment changes (such as environment variables defined) will persist until they log out

[xdelaruelle]

Hi @ernstki

Version 3.2.x provides the createmodule.sh tool that translates a shell script into the corresponding modulefile commands.

By calling that script into a modulefile, you mitigate the first potential issue you expose (if script is written for a different shell than user's login shell).

And with some specific code in your modulefile, you could mitigate the second issue you foresee, as described in https://github.com/easybuilders/easybuild-framework/issues/3275#issuecomment-613622228 proof of concept.

Which records in a specific environment variable the changes made by script, so when you unload modulefile you know what you have to unset.

[ernstki]

OK, thanks Xavier. The createmodule.sh script is not something I was aware of before. Great tip.

[rowanworth]

puts source /home/lsa/xilinx/Xilinx/Vitis/2019.2/settings64.sh

Modulefiles, as I understand them, eval their (stdout) output in the context of the current shell environment, and I'm pretty sure I've used this technique with success in the past.

The other issue with this approach (at least with 3.2.7) is that when loading multiple modules in the same command, environment changes made by the sourced script get clobbered if other modules change the same variables (because modules defers the environment changes until processing all module files, and doesn't account for the environment changes made during processing).

I hadn't heard of createmodule.sh before either and was halfway towards rolling my own, so I'm happy to find this thread :) Kind of wish it would accept a command instead of a script (eg. conda activate) but not hard to workaround.

[ernstki]

@rowanworth Yeah, to do that robustly, you'd almost need little wrapper functions written in shell script that add and remove entries from PATH-like variables (without creating duplicate entries). And at that point, why not just use Environment Modules? ;)

Bash's process substitution is your friend! I don't have conda available, but activating a virtualenv has got to be close, right?

$ createmodule.sh <(echo 'source venv/bin/activate')
#%Module 1.0
prepend-path    PATH    /path/to/venv/bin
setenv      PS1 (venv)
setenv      VIRTUAL_ENV /path/to/venv

If you do this a lot, you could wrap it in a function function in your ~/.bashrc to make it more convenient:

# assuming 'createmodule.sh' were in your search path somewhere
# usage: module-from-command command [args]
module-from-command() {
    createmodule.sh <( echo "$*" )
}

module-from-command source venv/bin/activate
# result: <same as above>

That's admittedly not being super-smart about whitespace in its arguments (createmodule.sh has problems of its own, noted from the value of PS1, which is wrong because it contains internal whitespace), but it's a start, anyway.

[rowanworth]

Yes, process substitution was the workaround I was referring to -- thanks!

It didn't quite suffice for my case; the conda activation ended up shuffling one or two path elements around (don't ask me why or how) which defeated the prepend detection logic in createmodule.sh. I think there is an argument for destructuring PATH/similar variables and diffing the elements, but that's way too big of a tangent for this closed issue ;)

[ernstki]

@rowanworth open a new issue for create_module.sh and "cc: @ernstki" at the bottom of the issue description.

I would never admit this in a public forum, but I think I might legitimately qualify as a "professional Bash programmer," and I'm curious what you've found, and whether there isn't a solution.

[marseaplage]

Hi

Could you help with the information about how can I modify a modulefile such that a set of environmental variables can be loaded when the user executes: “module load intel-mpicc/openfoam/10.0” since that program requires a set of variables be loaded that is called “bashrc”. I normally run it with: “source …/bashrc” I have installed:

Modules based on Lua: Version 8.5.22 2021-11-01 13:46 -05:00 by Robert McLay mclay@tacc.utexas.edu

My modulefiles is as follows: proc ModulesHelp { } {

puts stderr " " puts stderr "This module loads openfoam built with the intel compiler and mpicc" puts stderr "toolbook openfoam" puts stderr "\nVersion 10.0\n"

} module-whatis "Name: openfoam compiled with intel compiler and mpicc" module-whatis "Version: 10.0" module-whatis "Category: Toolbox" module-whatis "Description: openfoam compiled with intel compiler and mpicc" module-whatis "URL https://openfoam.org"

set version 10.0

prepend-path PATH /opt/ohpc/pub/libs/intel/mpicc/openfoam/10.0/OpenFOAM-10/bin prepend-path MANPATH /opt/ohpc/pub/libs/intel/mpicc/openfoam/10.0/OpenFOAM-10/doc prepend-path INCLUDE /opt/ohpc/pub/libs/intel/mpicc/openfoam/10.0/OpenFOAM-10/src prepend-path LD_LIBRARY_PATH /opt/ohpc/pub/libs/intel/mpicc/openfoam/10.0/OpenFOAM-10/etc

setenv OPENFOAM_DIR /opt/ohpc/pub/libs/intel/mpicc/openfoam/10.0/OpenFOAM-10 setenv OPENFOAM_BIN /opt/ohpc/pub/libs/intel/mpicc/openfoam/10.0/OpenFOAM-10/bin setenv OPENFOAM_LIB /opt/ohpc/pub/libs/intel/mpicc/openfoam/10.0/OpenFOAM-10/etc setenv OPENFOAM_INC /opt/ohpc/pub/libs/intel/mpicc/openfoam/10.0/OpenFOAM-10/src prepend-path MODULEPATH /opt/ohpc/pub/moduledeps/intel-mpicc

depends-on intel-oneapi depends-on mpi depends-on gnu9 depends-on openmpi4 depends-on compiler/2022.2.0

I am not able to update the lmod because it belongs to the ohpc 2.4 version and I do not want to lose compatibility.

Thank you in advance for your help.

[xdelaruelle]

@marseaplage Open the modulefile corresponding to the intel-mpicc/openfoam/10.0 module and add in this file the new variable definition (with setenv or prepend-path commands).

[marseaplage]

@marseaplage Open the modulefile corresponding to the intel-mpicc/openfoam/10.0 module and add in this file the new variable definition (with setenv or prepend-path commands).

Hi Xavier is not that simple. I need when the openfoam module is loaded to load a bashrc script of environment variables. Normally by cmd I just run the command: "source .../openfoam/bashrc". I need something similar to the "source_sh" option that exists in lmod 8.6 (I thin) but mine is 8.5 and I can not update it because I can lose ohpc compatibility. Could you please suggest me how can I achieve that?

[xdelaruelle]

Best thing you can do is to update Lmod on your system. I do not think you will loose ohpc compatibility by doing so. I do not know other means to proceed with Lmod <8.6. I know how to proceed with Modules (this project) without source-sh by using createmodules.sh script within a modulefile. With Lmod, maybe calling sh_to_modulefile.in.lua within modulefile may work.

[marseaplage]

Best thing you can do is to update Lmod on your system. I do not think you will loose ohpc compatibility by doing so. I do not know other means to proceed with Lmod <8.6. I know how to proceed with Modules (this project) without source-sh by using createmodules.sh script within a modulefile. With Lmod, maybe calling sh_to_modulefile.in.lua within modulefile may work.

I prefer not updating the lmod because the current version that I have integrated with ohpc only uses the 8.5 one. Could you please explain to me with more detail about how can I use the script sh_to_modulefile.in.lua to use it i my openfoam modulefile. I am new on this, I really appreciate your patience and help in advance.

[xdelaruelle]

I prefer not updating the lmod because the current version that I have integrated with ohpc only uses the 8.5 one. Could you please explain to me with more detail about how can I use the script sh_to_modulefile.in.lua to use it i my openfoam modulefile. I am new on this, I really appreciate your patience and help in advance.

I do not know, this is just a hint. I suggest you to ask such details on the Lmod project, you will find there people with more Lmod expertise than me :-).

[marseaplage]

I prefer not updating the lmod because the current version that I have integrated with ohpc only uses the 8.5 one. Could you please explain to me with more detail about how can I use the script sh_to_modulefile.in.lua to use it i my openfoam modulefile. I am new on this, I really appreciate your patience and help in advance.

I do not know, this is just a hint. I suggest you to ask such details on the Lmod project, you will find there people with more Lmod expertise than me :-).

Ok thank you so much four your suggestions :)