search

easybuilders / easybuild-easyblocks

Collection of easyblocks that implement support for building and installing software with EasyBuild.
https://easybuild.io
GNU General Public License v2.0
106 stars 285 forks source link

Enhance generic `Bundle` easyblock to transfer module requirements of components #3472

Closed Thyre closed 6 days ago

Thyre commented 1 month ago

This PR implements the make_module_req_guess method for the generic Bundle EasyBlock. With this, all the requirements of the components in a bundle are transferred correctly to the final module. Previously, this could lead to missing environment variables, letting the build succeed but still resulting in a broken module, for example because PATH is not set.

Fixes https://github.com/easybuilders/easybuild-easyblocks/issues/2733

Thyre commented 1 month ago

I've only tested the changes with https://github.com/easybuilders/easybuild-easyconfigs/pull/21582 so far. I'll try to gather the EasyConfigs using Bundle in some way to make sure that I did not break them, since this is a change on a widely used EasyBlock.

Unfortunately, I can only test GCC 12.3.0 and newer easily and are somewhat limited with my compute resources. So this might take a while..

Thyre commented 1 month ago

Example with the changed EasyBlock:

https://github.com/easybuilders/easybuild-easyconfigs/pull/21582#issuecomment-2397359586

Thyre commented 1 month ago

Test report by @Thyre

Overview of tested easyconfigs (in order)

Build succeeded for 1 out of 1 (1 easyconfigs in total) datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3 See https://gist.github.com/Thyre/a33ae670ab1932bdcfff62bae82c3f53 for a full test report.

Thyre commented 1 month ago

Test report by @Thyre

Overview of tested easyconfigs (in order)

Build succeeded for 25 out of 26 (7 easyconfigs in total) datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3 See https://gist.github.com/Thyre/3b8848b109d905611283a882db8bb378 for a full test report.

The failed one doesn't look related:

Building a mixed python/rust project
       Found pyo3 bindings
       maturin failed
        Caused by: Python interpreter should be a kind of interpreter (e.g. 'python3.8' or 'pypy3.9') when cross-compiling, got path to interpreter: /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.12.3-GCCcore-13.3.0/bin/python
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/eb-1tc417di/pip-modern-metadata-s3zbqxe6', '--interpreter', '/tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.12.3-GCCcore-13.3.0/bin/python']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /tmp/eb-1tc417di/pip-modern-metadata-s3zbqxe6 --interpreter /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.12.3-GCCcore-13.3.0/bin/python`
      [end of output]

The tested ones include normal Bundle (e.g. GCC & Autotools), PerlBundle (Perl-bundle-CPAN), PythonBundle (cffi) and JuliaBundle (Circuitscape). CargoPythonBundle failed due to the failure shown above.

Thyre commented 1 month ago

Changed PR so that the EasyBlock for each component is stored and not initialized in multiple steps.

Thyre commented 1 month ago

Based on https://docs.easybuild.io/version-specific/easyblocks/?h=bundle, this change should affect the following EasyBlocks:

I will not be able to test all of them (e.g. CrayToolchain), but I'll try to get one EasyConfig for all of these EasyBlocks to make sure that nothing broke and compare their modules.

Thyre commented 3 weeks ago

I've chosen a few EasyConfigs to compare with this MR and the current develop.

Building will take quite some time. I'll post the resulting module diffs later.

Thyre commented 2 weeks ago

Now that all modules are finally installed (but I had to partially select other modules due to general installation issues), I found one bug with the proposed changes:

When checking crypt4gh-1.7-GCC-12.3.0.eb, the module does a sanity check for an executable in /bin. This path is correctly added in develop:

help([==[

Description
===========
crypt4gh is a Python tool to encrypt, decrypt or re-encrypt files, 
according to the GA4GH encryption file format.

More information
================
 - Homepage: https://github.com/EGA-archive/crypt4gh

Included extensions
===================
crypt4gh-1.7
]==])

whatis([==[Description: crypt4gh is a Python tool to encrypt, decrypt or re-encrypt files, 
according to the GA4GH encryption file format.]==])
whatis([==[Homepage: https://github.com/EGA-archive/crypt4gh]==])
whatis([==[URL: https://github.com/EGA-archive/crypt4gh]==])
whatis([==[Extensions: crypt4gh-1.7]==])

local root = "/tank/Programs/Linux/EasyBuild/datenlager/Software/software/crypt4gh/1.7-GCC-12.3.0"

conflict("crypt4gh")

if not ( isloaded("GCC/12.3.0") ) then
    load("GCC/12.3.0")
end

if not ( isloaded("Python/3.11.3-GCCcore-12.3.0") ) then
    load("Python/3.11.3-GCCcore-12.3.0")
end

if not ( isloaded("Python-bundle-PyPI/2023.06-GCCcore-12.3.0") ) then
    load("Python-bundle-PyPI/2023.06-GCCcore-12.3.0")
end

if not ( isloaded("PyYAML/6.0-GCCcore-12.3.0") ) then
    load("PyYAML/6.0-GCCcore-12.3.0")
end

if not ( isloaded("cryptography/41.0.1-GCCcore-12.3.0") ) then
    load("cryptography/41.0.1-GCCcore-12.3.0")
end

if not ( isloaded("bcrypt/4.0.1-GCCcore-12.3.0") ) then
    load("bcrypt/4.0.1-GCCcore-12.3.0")
end

prepend_path("CMAKE_PREFIX_PATH", root)
prepend_path("LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("PATH", pathJoin(root, "bin"))
setenv("EBROOTCRYPT4GH", root)
setenv("EBVERSIONCRYPT4GH", "1.7")
setenv("EBDEVELCRYPT4GH", pathJoin(root, "easybuild/crypt4gh-1.7-GCC-12.3.0-easybuild-devel"))

prepend_path("PYTHONPATH", pathJoin(root, "lib/python3.11/site-packages"))
-- Built with EasyBuild version 4.9.5.dev0
setenv("EBEXTSLISTCRYPT4GH", "crypt4gh-1.7")

but this fails with the proposed changes. CMAKE_PREFIX_PATH, LIBRARY_PATH and PATH are missing. I suspect that adding make_module_req_guess to Bundle causes this to fail.

Thyre commented 2 weeks ago

Test report by @Thyre

Overview of tested easyconfigs (in order)

Build succeeded for 9 out of 9 (9 easyconfigs in total) datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3 See https://gist.github.com/Thyre/75aa472a40750fd854cf2d3ec2ede6e3 for a full test report.

I'm not able to test jiter-0.4.1-GCCcore-12.3.0.eb due to the following error caused by maturin:

  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

   Preparing metadata (pyproject.toml) did not run successfully.
   exit code: 1
  > [6 lines of output]
       Found pyo3 bindings
       maturin failed
        Caused by: Python interpreter should be a kind of interpreter (e.g. 'python3.8' or 'pypy3.9') when cross-compiling, got path to interpreter: /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.11.3-GCCcore-12.3.0/bin/python
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/eb-ibucduig/pip-modern-metadata-debc3bzs', '--interpreter', '/tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.11.3-GCCcore-12.3.0/bin/python']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /tmp/eb-ibucduig/pip-modern-metadata-debc3bzs --interpreter /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.11.3-GCCcore-12.3.0/bin/python`
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

 Encountered error while generating package metadata.
> See above for output.

I've also checked if the module files of the EasyConfigs above changed. They did not.

Thyre commented 2 weeks ago

Full (re-)installation

Test report by @Thyre

Overview of tested easyconfigs (in order)

Build succeeded for 9 out of 9 (9 easyconfigs in total) datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3 See https://gist.github.com/Thyre/bbc3468bc740b0c511c9f1e665d32910 for a full test report.

Thyre commented 2 weeks ago

Module only

Test report by @Thyre

Overview of tested easyconfigs (in order)

Build succeeded for 9 out of 9 (9 easyconfigs in total) datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3 See https://gist.github.com/Thyre/c78da935cdd54f14102e4d932295777e for a full test report.

Thyre commented 2 weeks ago

Testing a few of the Stages 2025 EasyConfigs using Bundle/PythonBundle/... of the JSC repo

Test report by @Thyre

Overview of tested easyconfigs (in order)

Build succeeded for 5 out of 5 (5 easyconfigs in total) jrlogin05.jureca - Linux Rocky Linux 9.4, x86_64, AMD EPYC 7742 64-Core Processor, 2 x NVIDIA Quadro RTX 8000, 560.35.03, Python 3.9.18 See https://gist.github.com/Thyre/81e26fbe93c25adabffd33e5664ddc0b for a full test report.

Thyre commented 1 week ago

Bi-weekly EasyBuild conf call 2024-11-06: While these changes continue to work with EasyBuild 5.0.x, the make_module_req_guess method will be deprecated.

My current idea would be to continue with this PR as it is. Once https://github.com/easybuilders/easybuild-framework/pull/4653 is merged, I'll work on porting these changes to the new method for EasyBuild 5.0.x.

Thyre commented 1 week ago

@boegelbot please test @ jsc-zen3 EB_ARGS="X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb" CORE_CNT=16

boegelbot commented 1 week ago

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3472 EB_ARGS="X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3472 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

Test results coming soon (I hope)...

*- notification for comment with ID 2464270943 processed* *Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).*
boegelbot commented 1 week ago

Test report by @boegelbot

Overview of tested easyconfigs (in order)

Build succeeded for 9 out of 9 (9 easyconfigs in total) jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18 See https://gist.github.com/boegelbot/f0ee3fa8737148ccb5fae372b743d256 for a full test report.

Thyre commented 1 week ago

@boegelbot please test @ jsc-zen3 EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb"

boegelbot commented 1 week ago

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3472 EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3472 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

Test results coming soon (I hope)...

*- notification for comment with ID 2464420612 processed* *Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).*
boegelbot commented 1 week ago

Test report by @boegelbot

Overview of tested easyconfigs (in order)

Build succeeded for 9 out of 9 (9 easyconfigs in total) jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18 See https://gist.github.com/boegelbot/b951173c327e4bb0f8b38422bdf91456 for a full test report.

Thyre commented 1 week ago

Thanks a lot for the suggestions. I'll add them to the commit in a moment 😄

Thyre commented 1 week ago

@boegelbot please test @ jsc-zen3 EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb"

boegelbot commented 1 week ago

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3472 EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3472 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

Test results coming soon (I hope)...

*- notification for comment with ID 2464855921 processed* *Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).*
boegelbot commented 1 week ago

Test report by @boegelbot

Overview of tested easyconfigs (in order)

Build succeeded for 9 out of 9 (9 easyconfigs in total) jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18 See https://gist.github.com/boegelbot/2df274b1d2731957ac6b0e5be8d36daf for a full test report.

lexming commented 1 week ago

@Thyre thanks for the quick update! do you want to add your name to the list of contributors to this easyblock? If you do, there is no need to send new tests again :slightly_smiling_face:

Thyre commented 1 week ago

@Thyre thanks for the quick update! do you want to add your name to the list of contributors to this easyblock?

Done. Took longer than I want to admit to find the authors comment 😅

If you do, there is no need to send new tests again 🙂

Agreed, changing the comment shouldn't break anything. Just wanted to make sure nothing broke during the last changes.

lexming commented 6 days ago

Merging, thanks @Thyre !

boegel commented 3 days ago

@Thyre @lexming I think the changes here may lead to an infinite loop, the easyconfigs test suite is failing due to generating too many open files, and it seems to coincide with this PR getting merged...

boegel commented 3 days ago

@Thyre @lexming I think the changes here may lead to an infinite loop, the easyconfigs test suite is failing due to generating too many open files, and it seems to coincide with this PR getting merged...

I've opened #3504 to revert the changes that were done here. Purely based on timing, it seems like the changes here are the culprit for the trouble we're currently seeing in the easyconfigs test suite...

lexming commented 3 days ago

@Thyre the changes in this PR are still alive in 5.0.x branch (https://github.com/easybuilders/easybuild-easyblocks/blob/5.0.x/easybuild/easyblocks/generic/bundle.py). We can focus on that branch for trobleshooting and fixing.

Thyre commented 3 days ago

@Thyre the changes in this PR are still alive in 5.0.x branch (https://github.com/easybuilders/easybuild-easyblocks/blob/5.0.x/easybuild/easyblocks/generic/bundle.py). We can focus on that branch for trobleshooting and fixing.

@lexming We found the issue, see https://github.com/easybuilders/easybuild-easyconfigs/issues/21841. It basically just boils down to loggers not being closed during the EasyConfig tests. I'll prepare an updated PR with a fix for this (and the comment regarding clang_aomp.py) tomorrow.