search

easybuilders / easybuild-framework

EasyBuild is a software installation framework in Python that allows you to install software in a structured and robust way.
https://easybuild.io
GNU General Public License v2.0
147 stars 198 forks source link

bootstrap_eb.py failed to run. #3202

Open hongyi-zhao opened 4 years ago

hongyi-zhao commented 4 years ago

Hi,

I use Ubuntu 19.10 and try to install easybuild with the bootstrap script. Failed with the following info:

$ export EASYBUILD_PREFIX=$HOME/.local/easybuild
$ python bootstrap_eb.py $EASYBUILD_PREFIX
[[INFO]] EasyBuild bootstrap script (version 20200203.01, MD5: fcb6314d4e0747db9c28a71f8bb2870c)
[[INFO]] Found Python 2.7.17 (default, Nov  7 2019, 10:07:09) ; [GCC 9.2.1 20191008]

[[INFO]] Installation prefix /home/werner/.local/easybuild
[[INFO]] Found module command '/usr/local/lmod/lmod/libexec/lmod' via $LMOD_CMD (Lmod), so using it.
[[INFO]] Suitable setuptools installation already found, skipping stage 0...

[[INFO]] +++ STAGE 1: installing EasyBuild in temporary dir with easy_install...

[[INFO]] running pre-install command 'easy_install --quiet --upgrade --prefix=/tmp/tmpWEH5H5/eb_stage1 vsc-install<0.11.4'
[[ERROR]] Running 'easy_install --quiet --upgrade --prefix=/tmp/tmpWEH5H5/eb_stage1 vsc-install<0.11.4' failed: error: Could not find suitable distribution for Requirement.parse('vsc-install<0.11.4')
Traceback (most recent call last):
  File "bootstrap_eb.py", line 380, in run_easy_install
    easy_install.main(args)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 2371, in main
    **kw
  File "/usr/lib/python2.7/dist-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python2.7/distutils/core.py", line 166, in setup
    raise SystemExit, "error: " + str(msg)
SystemExit: error: Could not find suitable distribution for Requirement.parse('vsc-install<0.11.4')

$ python3 bootstrap_eb.py $EASYBUILD_PREFIX
[[INFO]] EasyBuild bootstrap script (version 20200203.01, MD5: fcb6314d4e0747db9c28a71f8bb2870c)
[[INFO]] Found Python 3.7.5 (default, Nov 20 2019, 09:21:52) ; [GCC 9.2.1 20191008]

[[INFO]] Installation prefix /home/werner/.local/easybuild
[[INFO]] Found module command '/usr/local/lmod/lmod/libexec/lmod' via $LMOD_CMD (Lmod), so using it.
[[INFO]] Suitable setuptools installation already found, skipping stage 0...

[[INFO]] +++ STAGE 1: installing EasyBuild in temporary dir with easy_install...

[[INFO]] installing EasyBuild with 'easy_install --quiet --upgrade --prefix=/tmp/tmpqmcv1jpf/eb_stage1 easybuild>=4.0'

[[INFO]] Note: a 'SyntaxError' may be reported for the easybuild/tools/py2vs3/py2.py module.
You can safely ignore this message, it will not affect the functionality of the EasyBuild installation.

[[ERROR]] Running 'easy_install --quiet --upgrade --prefix=/tmp/tmpqmcv1jpf/eb_stage1 easybuild>=4.0' failed: error: Could not find suitable distribution for Requirement.parse('easybuild>=4.0')
Traceback (most recent call last):
  File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 443, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 698, in easy_install
    raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('easybuild>=4.0')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bootstrap_eb.py", line 380, in run_easy_install
    easy_install.main(args)
  File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 2371, in main
    **kw
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.7/distutils/core.py", line 163, in setup
    raise SystemExit("error: " + str(msg))
SystemExit: error: Could not find suitable distribution for Requirement.parse('easybuild>=4.0')

Any hints for this issue?

Regards

victorusu commented 4 years ago

I also ran into problems...

python2

$ python2 --version
Python 2.7.14
$ python bootstrap_eb.py $EASYBUILD_PREFIX
[[INFO]] EasyBuild bootstrap script (version 20200203.01, MD5: fcb6314d4e0747db9c28a71f8bb2870c)
[[INFO]] Found Python 2.7.14 (default, Nov 30 2017, 11:33:17) [GCC]

[[INFO]] Installation prefix /home/bla/.local/easybuild
[[INFO]] Found module command 'modulecmd' (EnvironmentModulesC), so using it.
[[INFO]] Suitable setuptools installation already found, skipping stage 0...

[[INFO]] +++ STAGE 1: installing EasyBuild in temporary dir with easy_install...

[[INFO]] running pre-install command 'easy_install --quiet --upgrade --prefix=/tmp/tmphNgQHK/eb_stage1 vsc-install<0.11.4'
[[INFO]] running pre-install command 'easy_install --quiet --upgrade --prefix=/tmp/tmphNgQHK/eb_stage1 vsc-base<2.9.0'
[[INFO]] installing EasyBuild with 'easy_install --quiet --upgrade --prefix=/tmp/tmphNgQHK/eb_stage1 easybuild'

[[INFO]] Note: a 'SyntaxError' may be reported for the easybuild/tools/py2vs3/py3.py module.
You can safely ignore this message, it will not affect the functionality of the EasyBuild installation.

  File "build/bdist.linux-x86_64/egg/easybuild/tools/py2vs3/py3.py", line 85
    class WrapperBase(object, metaclass=metaclass):
                                       ^
SyntaxError: invalid syntax

[[INFO]] running post install command 'easy_install --upgrade --prefix=/tmp/tmphNgQHK/eb_stage1 vsc-base<2.9.0'

[[INFO]] +++ STAGE 2: installing EasyBuild in /home/bla/.local/easybuild with EasyBuild from stage 1...

== temporary log file in case of crash /tmp/eb-QcNORS/easybuild-Zwtihr.log

WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment:
* xalt/2.7.24

This is not recommended since it may affect the installation procedure(s) performed by EasyBuild.

To make EasyBuild allow particular loaded modules, use the --allow-loaded-modules configuration option.
To specify action to take when loaded modules are detected, use --detect-loaded-modules={error,ignore,purge,unload,warn}.

See http://easybuild.readthedocs.io/en/latest/Detecting_loaded_modules.html for more information.

WARNING: Deprecated functionality, will no longer work in v5.0: Use of 'dummy' toolchain is deprecated, use 'system' toolchain instead; see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more information

== processing EasyBuild easyconfig /tmp/tmphNgQHK/EasyBuild-4.1.1.eb
== building and installing EasyBuild/4.1.1...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== FAILED: Installation ended unsuccessfully (build directory: /tmp/tmphNgQHK/EasyBuild/4.1.1/dummy-dummy): build failed (first 300 chars): cmd " /usr/bin/python setup.py install --prefix=/home/bla/.local/easybuild/software/EasyBuild/4.1.1 " exited with exit code 1 and output:
Installing version 4.1.1 (required versions: API >= 4, easyblocks >= 4.1)
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution optio (took 10 sec)
== Results of the build can be found in the log file(s) /tmp/eb-QcNORS/easybuild-EasyBuild-4.1.1-20200219.101457.uMBeP.log
Traceback (most recent call last):
  File "bootstrap_eb.py", line 1157, in <module>
    main()
  File "bootstrap_eb.py", line 956, in main
    stage2(tmpdir, templates, install_path, distribute_egg_dir, sourcepath)
  File "bootstrap_eb.py", line 814, in stage2
    easybuild_main()
  File "/tmp/tmphNgQHK/eb_stage1/lib/python2.7/site-packages/easybuild_framework-4.1.1-py2.7.egg/easybuild/main.py", line 469, in main
    ecs_with_res = build_and_install_software(ordered_ecs, init_session_state, exit_on_failure=exit_on_failure)
  File "/tmp/tmphNgQHK/eb_stage1/lib/python2.7/site-packages/easybuild_framework-4.1.1-py2.7.egg/easybuild/main.py", line 148, in build_and_install_software
    raise EasyBuildError(test_msg)
easybuild.tools.build_log.EasyBuildError: 'Build of /tmp/tmphNgQHK/EasyBuild-4.1.1.eb failed (err: \'build failed (first 300 chars): cmd " /usr/bin/python setup.py install --prefix=/home/bla/.local/easybuild/software/EasyBuild/4.1.1 " exited with exit code 1 and output:\\nInstalling version 4.1.1 (required versions: API >= 4, easyblocks >= 4.1)\\n/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution optio\')'

python3

$ python3 --version
Python 3.6.9
$ python3 bootstrap_eb.py $EASYBUILD_PREFIX
[[INFO]] EasyBuild bootstrap script (version 20200203.01, MD5: fcb6314d4e0747db9c28a71f8bb2870c)
[[INFO]] Found Python 3.6.9 (default, Sep 09 2019, 18:26:39) [GCC]

[[INFO]] Installation prefix /home/bla/.local/easybuild
[[INFO]] Found module command 'modulecmd' (EnvironmentModulesC), so using it.
[[INFO]] Suitable setuptools installation already found, skipping stage 0...

[[INFO]] +++ STAGE 1: installing EasyBuild in temporary dir with easy_install...

[[INFO]] installing EasyBuild with 'easy_install --quiet --upgrade --prefix=/tmp/tmp4sdni3ge/eb_stage1 easybuild>=4.0'

[[INFO]] Note: a 'SyntaxError' may be reported for the easybuild/tools/py2vs3/py2.py module.
You can safely ignore this message, it will not affect the functionality of the EasyBuild installation.

  File "build/bdist.linux-x86_64/egg/easybuild/tools/py2vs3/py2.py", line 67
    raise exception_class, message, traceback
                         ^
SyntaxError: invalid syntax

[[INFO]] +++ STAGE 2: installing EasyBuild in /home/bla/.local/easybuild with EasyBuild from stage 1...

== temporary log file in case of crash /tmp/eb-0lxn8hr4/easybuild-pdv0kowf.log

WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment:
* xalt/2.7.24

This is not recommended since it may affect the installation procedure(s) performed by EasyBuild.

To make EasyBuild allow particular loaded modules, use the --allow-loaded-modules configuration option.
To specify action to take when loaded modules are detected, use --detect-loaded-modules={error,ignore,purge,unload,warn}.

See http://easybuild.readthedocs.io/en/latest/Detecting_loaded_modules.html for more information.

WARNING: Deprecated functionality, will no longer work in v5.0: Use of 'dummy' toolchain is deprecated, use 'system' toolchain instead; see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more information

== processing EasyBuild easyconfig /tmp/tmp4sdni3ge/EasyBuild-4.1.1.eb
== building and installing EasyBuild/4.1.1...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== FAILED: Installation ended unsuccessfully (build directory: /tmp/tmp4sdni3ge/EasyBuild/4.1.1/dummy-dummy): build failed (first 300 chars): cmd " /usr/bin/python3 setup.py install --prefix=/home/bla/.local/easybuild/software/EasyBuild/4.1.1 " exited with exit code 1 and output:
Installing version 4.1.1 (required versions: API >= 4, easyblocks >= 4.1)
running install
running build
running install_data
creating /home/bla/.local/ (took 12 sec)
== Results of the build can be found in the log file(s) /tmp/eb-0lxn8hr4/easybuild-EasyBuild-4.1.1-20200219.101111.KRyjY.log
Traceback (most recent call last):
  File "bootstrap_eb.py", line 1157, in <module>
    main()
  File "bootstrap_eb.py", line 956, in main
    stage2(tmpdir, templates, install_path, distribute_egg_dir, sourcepath)
  File "bootstrap_eb.py", line 814, in stage2
    easybuild_main()
  File "/tmp/tmp4sdni3ge/eb_stage1/lib/python3.6/site-packages/easybuild_framework-4.1.1-py3.6.egg/easybuild/main.py", line 469, in main
    ecs_with_res = build_and_install_software(ordered_ecs, init_session_state, exit_on_failure=exit_on_failure)
  File "/tmp/tmp4sdni3ge/eb_stage1/lib/python3.6/site-packages/easybuild_framework-4.1.1-py3.6.egg/easybuild/main.py", line 148, in build_and_install_software
    raise EasyBuildError(test_msg)
easybuild.tools.build_log.EasyBuildError: 'Build of /tmp/tmp4sdni3ge/EasyBuild-4.1.1.eb failed (err: \'build failed (first 300 chars): cmd " /usr/bin/python3 setup.py install --prefix=/home/bla/.local/easybuild/software/EasyBuild/4.1.1 " exited with exit code 1 and output:\\nInstalling version 4.1.1 (required versions: API >= 4, easyblocks >= 4.1)\\nrunning install\\nrunning build\\nrunning install_data\\ncreating /home/bla/.local/\')'
Flamefire commented 4 years ago

There is indeed a failure on Python2 at least as PyPi no longer supports non-SSL connections as done by the old setuptools package used

hongyi-zhao commented 4 years ago

But, as you can see, it also failed with Python3.

Flamefire commented 4 years ago

Then we need the log of that as it is another issue

Flamefire commented 4 years ago

The original issue reported here by @hongyi-zhao will be fixed by #3211

The issue from @victorusu is something else and should go into a new issue with logs attached.

Note that Python3 support of the bootstrap is flaky and we need to come up with something better for the future

boegel commented 4 years ago

@victorusu Can you open a new issue for the bootstrap issue on Python 3, providing the full log (EASYBUILD_BOOTSTRAP_DEBUG=1 python bootstrap_eb.py $PREFIX 2>&1 | tee eb_bootstrap.log)?

victorusu commented 4 years ago

@boegel and @Flamefire, the issue that I saw is not related to EB. It is was a "quota disk exceed error". Sorry for not reporting back. I have created another issue related to that. #3200