search

easybuilders / easybuild

EasyBuild - building software with ease
http://easybuild.io
GNU General Public License v2.0
464 stars 143 forks source link

M4-1.4.18.eb failed - Changing environment as dictated by module failed #759

Closed kelbstf closed 2 years ago

kelbstf commented 2 years ago

Hello,

i have difficulties with M4 to install in a situation where EasyBuild 4.5.0 is installed via pip3 in a venv. All packages work well (M4 is part of the attempt to eb foss-2020a.eb), but M4 stumbles upon the LMod module of EasyBuild (the one i use to isntall), which activates the venv like this:

if { [ module-info mode load ] } {
    puts stdout "source /opt/ohpc/admin/easybuild/bin/activate;"
} elseif { [ module-info mode remove ] } {
    puts stdout "deactivate;"
}

This is what causes the compilation fail:

eb foss-2020a.eb --hidden --robot

== FAILED: Installation ended unsuccessfully 
(build directory: /opt/ohpc/admin/easybuild/easybuild/build/M4/1.4.18/system-system): build failed\
(first 300 chars): Changing environment as dictated by module failed: \
name 'source' is not defined (stdout: source /opt/ohpc/admin/easybuild/bin/activate;\
import os\
os.environ["__LMOD_REF_COUNT_CMAKE_PREFIX_PATH"] = \
"del os.environ["__LMOD_REF_COUNT_CMAKE_PREFIX_PATH"] \
os.environ["CMAKE_PREFIX_PATH"] = '' (took 22 secs)\

== Results of the build can be found in the log file(s) \
/tmp/eb-v5jfeihu/easybuild-M4-1.4.18-20211117.151856.blMoY.log

ERROR: Build of /opt/ohpc/admin/easybuild/easybuild/easyconfigs/m/M4/M4-1.4.18.eb failed (err: 'build failed (first 300 chars): 
Changing environment as dictated by module failed: 
name \'source\' is not defined 
(stdout: source /opt/ohpc/admin/easybuild/bin/activate;\nimport os\nos.environ["__LMOD_REF_COUNT_CMAKE_PREFIX_PATH"] = \'\'\ndel os.environ["__LMOD_REF_COUNT_CMAKE_PREFIX_PATH"]\nos.environ["CMAKE_PREFIX_PATH"] = \'\'')

Does it make sense that M4 actually attempts to load the module? All other packages seems not to depend on it... And if so: what might be the problem with source? On the CLI, it works perfectly fine...

kelbstf commented 2 years ago

Hello. i found this previous discussion: https://github.com/easybuilders/easybuild-framework/issues/3275

I am not sure, whether this is actually the same situation, because i try to just source a file, thus the file is currently not executable. LMod seems to support s.th. similar: https://lmod.readthedocs.io/en/latest/260_sh_to_modulefile.html#calling-the-shell-script-directly-inside-a-modulefile", but apparently only for 8.6+, while my version is "8.2.10".

What i still don't understand:

  1. if i load the module via module load on the CLI, the sourced venv is perfectly established (using the TCL "source ..." syntax; not the official LMod "source_sh ()" method). But when Easybuild tests the loadting of the M4 module, it seems to fail...
  2. M4 ssems to be the only package affected so far, all other packages are succesfully compiled.

Does anyone has an idea?

boegel commented 2 years ago

@kelbstf This module file has the source statement in it exactly, and where does that module file come from?

The problem here is that EasyBuild is consuming module files via Python, so loading the module is done via $LMOD_CMD python load (whereas loading the module manually via module load is equivalent to loading it via $LMOD_CMD bash load). The source command is not valid Python syntax, which explains the error you are getting.

To help you out, we need more info on where the module file using source comes from.

You'll basically need to make sure that EasyBuild doesn't try to load that module (it's not clear to me why it would, currently, but I don't have enough info about this module file this is exactly).

kelbstf commented 2 years ago

Hello @boegel,

thank you very much for looking into it! And for explaining how the module file is parsed in EasyBuild.

If i understand you right, there seems to be 2 aspects to this problem:

  1. why the compilation of M4 triggers loading the EasyBild module in the first place (while it actually is loaded at that time)
  2. how the venv call can be made accessible to python

@Background/use case: "source" is meant to launch the python venv any time but only for the occasion of using EasyBuild to maintain software. This seemed the most convenient and "failsafe" (prerequisite venv can't be missed when launching EasyBuild), non-global solution, where the same account still should have options to launch alternative venvs outside of using EasyBuild.

The Lmod (8.2) module for EasyBuild is written by me, as i installed EasyBuild via pip, not via EasyBuild:

/opt/ohpc/admin/easybuild/easybuild/modulefiles/EasyBuild/4.5.0
#%Module1.0#####################################################################

proc ModulesHelp { } {
puts stderr "Module for loading easybuild"
puts stderr "\nVersion 4.5.0\n"
}
module-whatis "Name: easybuild"
module-whatis "Version: 4.5.0"
module-whatis "Category: system tool"
module-whatis "Description: Software build and installation framework"
module-whatis "URL: https://easybuilders.github.io/easybuild/"
set             version                 4.5.0
set             home                    $::env(HOME)

if { [ module-info mode load ] } {
    puts stdout "source /opt/ohpc/admin/easybuild/bin/activate;"
} elseif { [ module-info mode remove ] } {
    puts stdout "deactivate;"
}

source_sh("python", "/opt/ohpc/admin/easybuild/bin/activate activate")

prepend-path    PATH                    /opt/ohpc/admin/easybuild/bin
prepend-path    PATH                    /opt/ohpc/admin/lmod/lmod/libexec
prepend-path    LD_LIBRARY_PATH         /opt/ohpc/admin/easybuild/lib
prepend-path    LIBRARY_PATH            /opt/ohpc/admin/easybuild/lib
module          use                     /opt/ohpc/pub/moduleseb
setenv          EASYBUILD_CONFIGFILES   /opt/ohpc/admin/easybuild/etc/config.cfg
setenv          EBROOTEASYBUILD         /opt/ohpc/admin/easybuild
setenv          EBVERSIONEASYBUILD      4.5.0
setenv          EB_PYTHON               python3
setenv          EASYBUILD_MODULE_NAMING_SCHEME  CategorizedHMNS
prepend-path    PYTHONPATH             /opt/ohpc/admin/easybuild/lib/python3.6/site-packages

#source_sh("python", "/opt/ohpc/admin/easybuild/bin/activate activate")

For the python access i now tried with what LMod proposes for accessing scripts in LMod module (),, using the "source_sh" statement above.

But that fails with:

Lmod has detected the following error: 
/opt/ohpc/admin/easybuild/easybuild/modulefiles/EasyBuild/4.5.0: 
(EasyBuild/4.5.0): extra characters after close-quote 
While processing the following module(s):
    Module fullname     Module Filename
    ---------------     ---------------
    EasyBuild/4.5.0  /opt/ohpc/admin/easybuild/easybuild/modulefiles/EasyBuild/4.5.0

This might still be due to "Lmod 8.2.10 (TCL Version 8.6.8)", while apparently source_sh depends on LMod 8.6+. Unfortunately i currently don't have access to LMod 8.6+ in the very short term :-/

But in case the solution would depend on Lmod 8.6+, it would still be interesting whether this would be possible in general? The general relevance could be: "Running EasyBuild in a temporary venv".

Best regards

kelbstf commented 2 years ago

Hello, i just wanted to update that in an effort to keep the implementation simple, i believe it's better to remove the loading of the venv should be done externally from the module. At least switching to Lua to begin with would possibly be the better choice for s.th. like this. From my point of view i then would close this issue, if no one disagrees?