search

easybuilders / easybuild-framework

EasyBuild is a software installation framework in Python that allows you to install software in a structured and robust way.
https://easybuild.io
GNU General Public License v2.0
152 stars 202 forks source link

Trying to install EasyBuild 4.0.0 with (re)bootstrap #3035

Open eylenth opened 5 years ago

eylenth commented 5 years ago

Hi

I want to upgrade EasyBuild to version 4.0.0 with the re-bootstrap procedure But I am getting the following error when I try to upgrade with Python2

$ pwd
/tools/eb
$ export EASYBUILD_BOOTSTRAP_SOURCEPATH=/tools/eb/src/
$ /usr/bin/python2 src/bootstrap_eb.py /tools/eb
[[INFO]] EasyBuild bootstrap script (version 20190922.01, MD5: 039b743a44cde1549cfc0f15796d4dcd)
[[INFO]] Found Python 2.7.5 (default, May 31 2018, 09:41:32) ; [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

[[INFO]] Installation prefix /tools/eb
[[INFO]] Fetching sources from /tools/eb/src/...
[[INFO]] Found module command '/usr/share/lmod/lmod/libexec/lmod' via $LMOD_CMD (Lmod), so using it.
[[INFO]] Suitable setuptools installation already found, skipping stage 0...

[[INFO]] +++ STAGE 1: installing EasyBuild in temporary dir with easy_install...

[[INFO]] Fetching sources from /tools/eb/src/...
[[INFO]] Found /tools/eb/src/easybuild-framework-4.0.0.tar.gz for easybuild-framework package
[[INFO]] Found /tools/eb/src/easybuild-easyblocks-4.0.0.tar.gz for easybuild-easyblocks package
[[INFO]] Found /tools/eb/src/easybuild-easyconfigs-4.0.0.tar.gz for easybuild-easyconfigs package
[[INFO]] running pre-install command 'easy_install --quiet --upgrade --prefix=/tmp/tmpgBD466/eb_stage1 /tools/eb/src/easybuild-framework-4.0.0.tar.gz /tools/eb/src/easybuild-easyblocks-4.0.0.tar.gz vsc-install<0.11.4'
  File "build/bdist.linux-x86_64/egg/easybuild/tools/py2vs3/py3.py", line 82
    class WrapperBase(object, metaclass=metaclass):
                                       ^
SyntaxError: invalid syntax

[[INFO]] running pre-install command 'easy_install --quiet --upgrade --prefix=/tmp/tmpgBD466/eb_stage1 /tools/eb/src/easybuild-framework-4.0.0.tar.gz /tools/eb/src/easybuild-easyblocks-4.0.0.tar.gz vsc-base<2.9.0'
  File "build/bdist.linux-x86_64/egg/easybuild/tools/py2vs3/py3.py", line 82
    class WrapperBase(object, metaclass=metaclass):
                                       ^
SyntaxError: invalid syntax

[[INFO]] installing EasyBuild with 'easy_install --quiet --upgrade --prefix=/tmp/tmpgBD466/eb_stage1 /tools/eb/src/easybuild-framework-4.0.0.tar.gz /tools/eb/src/easybuild-easyblocks-4.0.0.tar.gz /tools/eb/src/easybuild-easyconfigs-4.0.0.tar.gz'

[[INFO]] Note: a 'SyntaxError' may be reported for the easybuild/tools/py2vs3/py3.py module.
You can safely ignore this message, it will not affect the functionality of the EasyBuild installation.

  File "build/bdist.linux-x86_64/egg/easybuild/tools/py2vs3/py3.py", line 82
    class WrapperBase(object, metaclass=metaclass):
                                       ^
SyntaxError: invalid syntax

 $ python
Python 2.7.5 (default, May 31 2018, 09:41:32) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.version_info[0]
2

for some reason it is trying to use easybuild/tools/py2vs3/py3.py instead of easybuild/tools/py2vs3/py2.py in STAGE 1 .

boegel commented 5 years ago

@eylenth Can you try without setting $EASYBUILD_BOOTSTRAP_SOURCEPATH (so the bootstrap script will download the source tarballs from PyPI)?

You're definitely hitting a bug in the bootstrap script, the logic is a bit funky, it shouldn't be trying to install vsc-install or vsc-base at all anymore, since they are no longer required for EasyBuild v4.0.0.

Any reason you're not using eb --install-latest-eb-release with an existing EasyBuild installation instead?

eylenth commented 5 years ago

It seems to work now...

$ /usr/bin/python2 src/bootstrap_eb.py /tools/eb
[[INFO]] EasyBuild bootstrap script (version 20190922.01, MD5: 039b743a44cde1549cfc0f15796d4dcd)
[[INFO]] Found Python 2.7.5 (default, May 31 2018, 09:41:32) ; [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

[[INFO]] Installation prefix /tools/eb
[[INFO]] Found module command '/usr/share/lmod/lmod/libexec/lmod' via $LMOD_CMD (Lmod), so using it.
[[INFO]] Suitable setuptools installation already found, skipping stage 0...

[[INFO]] +++ STAGE 1: installing EasyBuild in temporary dir with easy_install...

[[INFO]] running pre-install command 'easy_install --quiet --upgrade --prefix=/tmp/tmpqhDolz/eb_stage1 vsc-install<0.11.4'
[[INFO]] running pre-install command 'easy_install --quiet --upgrade --prefix=/tmp/tmpqhDolz/eb_stage1 vsc-base<2.9.0'
[[INFO]] installing EasyBuild with 'easy_install --quiet --upgrade --prefix=/tmp/tmpqhDolz/eb_stage1 easybuild'

[[INFO]] Note: a 'SyntaxError' may be reported for the easybuild/tools/py2vs3/py3.py module.
You can safely ignore this message, it will not affect the functionality of the EasyBuild installation.

  File "build/bdist.linux-x86_64/egg/easybuild/tools/py2vs3/py3.py", line 82
    class WrapperBase(object, metaclass=metaclass):
                                       ^
SyntaxError: invalid syntax

[[INFO]] running post install command 'easy_install --upgrade --prefix=/tmp/tmpqhDolz/eb_stage1 vsc-base<2.9.0'

[[INFO]] +++ STAGE 2: installing EasyBuild in /tools/eb with EasyBuild from stage 1...

== temporary log file in case of crash /tmp/eb-U36rxa/easybuild-8QHYI9.log

WARNING: Deprecated functionality, will no longer work in v5.0: Use of 'dummy' toolchain is deprecated, use 'system' toolchain instead; see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more information

== processing EasyBuild easyconfig /tmp/tmpqhDolz/EasyBuild-4.0.0.eb
== building and installing Core/EasyBuild/4.0.0...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== restore after iterating...
== postprocessing...
== sanity checking...
== cleaning up...
== creating module...
== permissions...
== packaging...
== COMPLETED: Installation ended successfully (took 46 sec)
== Results of the build can be found in the log file(s) /tools/eb/software/EasyBuild/4.0.0/easybuild/easybuild-EasyBuild-4.0.0-20190930.112809.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-U36rxa/easybuild-8QHYI9.log* have been removed.
== Temporary directory /tmp/eb-U36rxa has been removed.
[[INFO]] EasyBuild module installed: /tools/eb/modules/all/Core/EasyBuild/4.0.0.lua

[[INFO]] Bootstrapping EasyBuild completed!

[[INFO]] EasyBuild v4.0.0 was installed to /tools/eb, so make sure your $MODULEPATH includes /tools/eb/modules/all

[[INFO]] Run 'module load EasyBuild', and run 'eb --help' to get help on using EasyBuild.
[[INFO]] Set $EASYBUILD_MODULES_TOOL to 'Lmod' to use the same modules tool as was used now.

[[INFO]] By default, EasyBuild will install software to $HOME/.local/easybuild.
[[INFO]] To install software with EasyBuild to /tools/eb, set $EASYBUILD_INSTALLPATH accordingly.
[[INFO]] See http://easybuild.readthedocs.org/en/latest/Configuration.html for details on configuring EasyBuild

But I still see the warning message about py3.py

The reason, why I am upgrading easybuild by rebootstrap, is because our installation/upgrade procedure is managed by the configuration management tool(puppet). It will be difficult to implement eb --install-latest-eb-release in the puppet code. And I am also injecting a customized module naming scheme into the tarball of easybuild-framework-4.0.0.tar.gz

boegel commented 5 years ago

@eylenth Good to hear you got it installed.

For the module naming scheme, you should park that somewhere and configure EasyBuild using --include-module-naming-schemes to pick up on it, rather than adding it to the framework installation...

eylenth commented 5 years ago

I would like to park it in a location configured with include-module-naming-schemes. But when I am rebootstrapping easybuild, it is not aware of $HOME/.config/easybuild/config.cfg (I think ?), so easybuild itself cannot be installed in the installation directory that is being configured with the customized module naming scheme.

So that's the reason why I am injecting that one customized module naming scheme in tarball of easyblock-framework instead of using --include-module-naming-schemes. But I don't mind to inject this. If the location of the module naming schemes will ever change, then I will adapt my build-pipeline.