trz42
commented
8 months ago First complete pass (except for some easystack files) is done. Remaining todos:
-
do second pass on current differences
- [ ]git diff nessi_2023.06 2023.06-software.eessi.io.github/workflows/test-pilot.nessi.no.ymlonly differences due to repo naming, repo config, supported CPU architectures --> ✅- [ ]diff
```diff diff --git a/.github/workflows/test-pilot.nessi.no.yml b/.github/workflows/test-software.eessi.io.yml similarity index 76% rename from .github/workflows/test-pilot.nessi.no.yml rename to .github/workflows/test-software.eessi.io.yml index 6342d5d..3ac341a 100644 --- a/.github/workflows/test-pilot.nessi.no.yml +++ b/.github/workflows/test-software.eessi.io.yml @@ -1,5 +1,5 @@ # documentation: https://help.github.com/en/articles/workflow-syntax-for-github-actions -name: Check for missing software installations in pilot.nessi.no +name: Check for missing software installations in software.eessi.io on: [push, pull_request, workflow_dispatch] permissions: contents: read # to fetch code (actions/checkout) @@ -12,37 +12,40 @@ jobs: EESSI_VERSION: - 2023.06 EESSI_SOFTWARE_SUBDIR_OVERRIDE: -# - aarch64/generic + - aarch64/generic + - aarch64/neoverse_n1 + - aarch64/neoverse_v1 - x86_64/amd/zen2 - - x86_64/intel/broadwell + - x86_64/amd/zen3 + - x86_64/intel/haswell - x86_64/intel/skylake_avx512 - x86_64/generic steps: - name: Check out software-layer repository uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0 - - name: Mount NESSI CernVM-FS repository + - name: Mount EESSI CernVM-FS pilot repository uses: cvmfs-contrib/github-action-cvmfs@d4641d0d591c9a5c3be23835ced2fb648b44c04b # v3.1 with: - cvmfs_config_package: https://github.com/NorESSI/filesystem-layer/releases/download/latest/cvmfs-config-nessi_latest_all.deb + cvmfs_config_package: https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb cvmfs_http_proxy: DIRECT - cvmfs_repositories: pilot.nessi.no + cvmfs_repositories: software.eessi.io - name: Test check_missing_installations.sh script run: | export EESSI_SOFTWARE_SUBDIR_OVERRIDE=${{matrix.EESSI_SOFTWARE_SUBDIR_OVERRIDE}} - source /cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}}/init/bash + source /cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}/init/bash # set $EESSI_CPU_FAMILY to the CPU architecture that corresponds to $EESSI_SOFTWARE_SUBDIR_OVERRIDE (part before the first slash), # to prevent issues with checks in the Easybuild configuration that use this variable export EESSI_CPU_FAMILY=${EESSI_SOFTWARE_SUBDIR_OVERRIDE%%/*} module load EasyBuild which eb eb --version - export EESSI_PREFIX=/cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}} + export EESSI_PREFIX=/cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}} export EESSI_OS_TYPE=linux env | grep ^EESSI | sort - echo "just run check_missing_installations.sh (should use easystacks/pilot.nessi.no/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-*.yml)" - for easystack_file in $(ls easystacks/pilot.nessi.no/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-eb-*.yml); do + echo "just run check_missing_installations.sh (should use easystacks/software.eessi.io/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-*.yml)" + for easystack_file in $(ls easystacks/software.eessi.io/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-eb-*.yml); do echo "check missing installations for ${easystack_file}..." ./check_missing_installations.sh ${easystack_file} ec=$? @@ -52,14 +55,14 @@ jobs: - name: Test check_missing_installations.sh with missing package (GCC/8.3.0) run: | export EESSI_SOFTWARE_SUBDIR_OVERRIDE=${{matrix.EESSI_SOFTWARE_SUBDIR_OVERRIDE}} - source /cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}}/init/bash + source /cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}/init/bash # set $EESSI_CPU_FAMILY to the CPU architecture that corresponds to $EESSI_SOFTWARE_SUBDIR_OVERRIDE (part before the first slash), # to prevent issues with checks in the Easybuild configuration that use this variable export EESSI_CPU_FAMILY=${EESSI_SOFTWARE_SUBDIR_OVERRIDE%%/*} module load EasyBuild which eb eb --version - export EESSI_PREFIX=/cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}} + export EESSI_PREFIX=/cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}} export EESSI_OS_TYPE=linux env | grep ^EESSI | sort # create dummy easystack file with a single entry (something that is not installed in EESSI) ```.github/workflows/test_eessi_container_script.yml- [ ]diff
```diff diff --git a/.github/workflows/test_eessi_container_script.yml b/.github/workflows/test_eessi_container_script.yml index 3bb67b4..33122e6 100644 --- a/.github/workflows/test_eessi_container_script.yml +++ b/.github/workflows/test_eessi_container_script.yml @@ -90,15 +90,15 @@ jobs: elif [[ ${{matrix.SCRIPT_TEST}} == 'readwrite' ]]; then outfile=out_readwrite.txt fn="test_${RANDOM}.txt" - echo "touch /cvmfs/pilot.nessi.no/${fn}" > test_script.sh + echo "touch /cvmfs/software.eessi.io/${fn}" > test_script.sh chmod u+x test_script.sh export SINGULARITY_BIND="$PWD:/test" ./eessi_container.sh --verbose --access rw --mode run /test/test_script.sh > ${outfile} tmpdir=$(grep "\-\-resume" ${outfile} | sed "s/.*--resume \([^']*\).*/\1/g") # note: must use '--access rw' again here, since touched file is in overlay upper dir - ./eessi_container.sh --verbose --resume ${tmpdir} --access rw --mode shell <<< "ls -l /cvmfs/pilot.nessi.no/${fn}" > ${outfile} - grep "/cvmfs/pilot.nessi.no/${fn}$" $outfile + ./eessi_container.sh --verbose --resume ${tmpdir} --access rw --mode shell <<< "ls -l /cvmfs/software.eessi.io/${fn}" > ${outfile} + grep "/cvmfs/software.eessi.io/${fn}$" $outfile # test use of --resume elif [[ ${{matrix.SCRIPT_TEST}} == 'resume' ]]; then @@ -120,12 +120,12 @@ jobs: elif [[ ${{matrix.SCRIPT_TEST}} == 'save' ]]; then outfile=out_save.txt fn="test_${RANDOM}.txt" - test_cmd="touch /cvmfs/pilot.nessi.no/${fn}" + test_cmd="touch /cvmfs/software.eessi.io/${fn}" ./eessi_container.sh --verbose --mode shell --access rw --save test-save.tar <<< "${test_cmd}" 2>&1 | tee ${outfile} rm -f ${outfile} - ./eessi_container.sh --verbose --mode shell --access rw --resume test-save.tar <<< "ls -l /cvmfs/pilot.nessi.no/${fn}" > ${outfile} - grep "/cvmfs/pilot.nessi.no/${fn}$" $outfile + ./eessi_container.sh --verbose --mode shell --access rw --resume test-save.tar <<< "ls -l /cvmfs/software.eessi.io/${fn}" > ${outfile} + grep "/cvmfs/software.eessi.io/${fn}$" $outfile tar tfv test-save.tar | grep "overlay-upper/${fn}" ```.github/workflows/tests_archdetect.yml- [ ]diff
```diff diff --git a/.github/workflows/tests_archdetect.yml b/.github/workflows/tests_archdetect.yml index 74dbf03..1e8b830 100644 --- a/.github/workflows/tests_archdetect.yml +++ b/.github/workflows/tests_archdetect.yml @@ -13,23 +13,23 @@ jobs: - x86_64/intel/skylake_avx512/archspec-linux-6132 - x86_64/amd/zen2/Azure-CentOS7-7V12 - x86_64/amd/zen3/Azure-CentOS7-7V73X - # commented out since these targets are currently not supported in pilot.nessi.no repo + - aarch64/neoverse_n1/Azure-Ubuntu20-Altra + - aarch64/neoverse_n1/AWS-awslinux-graviton2 + - aarch64/neoverse_v1/AWS-awslinux-graviton3 + # commented out since these targets are currently not supported in software.eessi.io repo # (and some tests assume that the corresponding subdirectory in software layer is there) # - ppc64le/power9le/unknown-power9le - # - aarch64/neoverse-n1/Azure-Ubuntu20-Altra - # - aarch64/neoverse-n1/AWS-awslinux-graviton2 - # - aarch64/neoverse-v1/AWS-awslinux-graviton3 fail-fast: false steps: - name: checkout uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0 - - name: Mount NESSI CernVM-FS repository + - name: Mount EESSI CernVM-FS pilot repository uses: cvmfs-contrib/github-action-cvmfs@d4641d0d591c9a5c3be23835ced2fb648b44c04b # v3.1 with: - cvmfs_config_package: https://github.com/NorESSI/filesystem-layer/releases/download/latest/cvmfs-config-nessi_latest_all.deb + cvmfs_config_package: https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb cvmfs_http_proxy: DIRECT - cvmfs_repositories: pilot.nessi.no + cvmfs_repositories: software.eessi.io - name: test eessi_archdetect.sh run: | ```EESSI-install-software.sh- [ ]diff
```diff diff --git a/EESSI-install-software.sh b/EESSI-install-software.sh index eb616b1..69de9d1 100755 --- a/EESSI-install-software.sh +++ b/EESSI-install-software.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Script to install NESSI software stack (version set through init/eessi_defaults) +# Script to install EESSI software stack (version set through init/eessi_defaults) # see example parsing of command line arguments at # https://wiki.bash-hackers.org/scripting/posparams#using_a_while_loop ```README.md- [ ]diff
```diff diff --git a/README.md b/README.md index 4f06139..ab73ad7 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,25 @@ # Software layer -The software layer of the NESSI project uses [EasyBuild](https://docs.easybuild.io), [Lmod](https://lmod.readthedocs.io) and [archspec](https://archspec.readthedocs.io). +The software layer of the EESSI project uses [EasyBuild](https://docs.easybuild.io), [Lmod](https://lmod.readthedocs.io) and [archspec](https://archspec.readthedocs.io). See also https://www.eessi.io/docs/software_layer . -## Accessing the software stack +## Pilot software stack You can set up your environment by sourcing the init script: ``` -$ source /cvmfs/pilot.nessi.no/versions/2023.06/init/bash -Found NESSI repo @ /cvmfs/pilot.nessi.no/versions/2023.06! -Derived subdirectory for software layer: x86_64/intel/broadwell +$ source /cvmfs/software.eessi.io/versions/2023.06/init/bash +Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06! +Derived subdirectory for software layer: x86_64/intel/haswell Using x86_64/intel/haswell subdirectory for software layer Initializing Lmod... -Prepending /cvmfs/pilot.nessi.no/versions/2023.06/software/x86_64/intel/broadwell/modules/all to $MODULEPATH... -Environment set up to use NESSI software stack, have fun! -{NESSI 2023.06} $ +Prepending /cvmfs/software.eessi.io/versions/2023.06/software/x86_64/intel/haswell/modules/all to $MODULEPATH... +Environment set up to use EESSI (2023.06), have fun! +[EESSI 2023.06] $ ``` -### Accessing NESSI via a container +### Accessing EESSI via a container You need Singularity version 3.7 or newer. Then, simply run @@ -40,6 +40,6 @@ Note, not all features/arguments listed via `./eessi_container.sh --help` are im The software in this repository is distributed under the terms of the [GNU General Public License v2.0](https://opensource.org/licenses/GPL-2.0). -See [LICENSE](https://github.com/NorESSI/software-layer/blob/main/LICENSE) for more information. +See [LICENSE](https://github.com/EESSI/software-layer/blob/main/LICENSE) for more information. SPDX-License-Identifier: GPL-2.0-only ```bot/build.sh- [ ]diff
```diff diff --git a/bot/build.sh b/bot/build.sh index 7eb15f3..1622e75 100755 --- a/bot/build.sh +++ b/bot/build.sh @@ -128,7 +128,7 @@ EESSI_REPOS_CFG_DIR_OVERRIDE=$(cfg_get_value "repository" "repos_cfg_dir") export EESSI_REPOS_CFG_DIR_OVERRIDE=${EESSI_REPOS_CFG_DIR_OVERRIDE:-${PWD}/cfg} echo "bot/build.sh: EESSI_REPOS_CFG_DIR_OVERRIDE='${EESSI_REPOS_CFG_DIR_OVERRIDE}'" -# determine NESSI version to be used from .repository.repo_version in ${JOB_CFG_FILE} +# determine EESSI version to be used from .repository.repo_version in ${JOB_CFG_FILE} # here, just set & export EESSI_VERSION_OVERRIDE # next script (eessi_container.sh) makes use of it via sourcing init scripts # (e.g., init/eessi_defaults or init/minimal_eessi_env) ```build_container.sh- [ ]diff
```diff diff --git a/build_container.sh b/build_container.sh new file mode 100755 index 0000000..23a9e66 --- /dev/null +++ b/build_container.sh @@ -0,0 +1,69 @@ +#!/bin/bash + +base_dir=$(dirname $(realpath $0)) + +BUILD_CONTAINER="docker://ghcr.io/eessi/build-node:debian11" + +if [ $# -lt 2 ]; then + echo "Usage: $0" >&2 + exit 1 +fi +SHELL_OR_RUN=$1 +EESSI_TMPDIR=$2 +shift 2 + +if [ "$SHELL_OR_RUN" == "run" ] && [ $# -eq 0 ]; then + echo "ERROR: No command specified to run?!" >&2 + exit 1 +fi + +# make sure specified temporary directory exists +mkdir -p $EESSI_TMPDIR + +echo "Using $EESSI_TMPDIR as parent for temporary directories..." + +# create temporary directories +mkdir -p $EESSI_TMPDIR/{home,overlay-upper,overlay-work} +mkdir -p $EESSI_TMPDIR/{var-lib-cvmfs,var-run-cvmfs} +# configure Singularity +export SINGULARITY_CACHEDIR=$EESSI_TMPDIR/singularity_cache + +# take into account that $SINGULARITY_BIND may be defined already, to bind additional paths into the build container +BIND_PATHS="$EESSI_TMPDIR/var-run-cvmfs:/var/run/cvmfs,$EESSI_TMPDIR/var-lib-cvmfs:/var/lib/cvmfs,$EESSI_TMPDIR" +if [ -z $SINGULARITY_BIND ]; then + export SINGULARITY_BIND="$BIND_PATHS" +else + export SINGULARITY_BIND="$SINGULARITY_BIND,$BIND_PATHS" +fi + +# allow that SINGULARITY_HOME is defined before script is run +if [ -z $SINGULARITY_HOME ]; then + export SINGULARITY_HOME="$EESSI_TMPDIR/home:/home/$USER" +fi + +source ${base_dir}/init/eessi_defaults +# strip "/cvmfs/" from default setting +repo_name=${EESSI_CVMFS_REPO/\/cvmfs\//} + +# set environment variables for fuse mounts in Singularity container +export EESSI_PILOT_READONLY="container:cvmfs2 ${repo_name} /cvmfs_ro/${repo_name}" +export EESSI_PILOT_WRITABLE_OVERLAY="container:fuse-overlayfs -o lowerdir=/cvmfs_ro/${repo_name} -o upperdir=$EESSI_TMPDIR/overlay-upper -o workdir=$EESSI_TMPDIR/overlay-work ${EESSI_CVMFS_REPO}" + +# pass $EESSI_SOFTWARE_SUBDIR_OVERRIDE into build container (if set) +if [ ! -z ${EESSI_SOFTWARE_SUBDIR_OVERRIDE} ]; then + export SINGULARITYENV_EESSI_SOFTWARE_SUBDIR_OVERRIDE=${EESSI_SOFTWARE_SUBDIR_OVERRIDE} + # also specify via $APPTAINERENV_* (future proof, cfr. https://apptainer.org/docs/user/latest/singularity_compatibility.html#singularity-environment-variable-compatibility) + export APPTAINERENV_EESSI_SOFTWARE_SUBDIR_OVERRIDE=${EESSI_SOFTWARE_SUBDIR_OVERRIDE} +fi + +if [ "$SHELL_OR_RUN" == "shell" ]; then + # start shell in Singularity container, with EESSI repository mounted with writable overlay + echo "Starting Singularity build container..." + singularity shell --fusemount "$EESSI_PILOT_READONLY" --fusemount "$EESSI_PILOT_WRITABLE_OVERLAY" $BUILD_CONTAINER +elif [ "$SHELL_OR_RUN" == "run" ]; then + echo "Running '$@' in Singularity build container..." + singularity exec --fusemount "$EESSI_PILOT_READONLY" --fusemount "$EESSI_PILOT_WRITABLE_OVERLAY" $BUILD_CONTAINER "$@" +else + echo "ERROR: Unknown action specified: $SHELL_OR_RUN (should be either 'shell' or 'run')" >&2 + exit 1 +fi ``` check_missing_installations.sh- [ ]diff
```diff diff --git a/check_missing_installations.sh b/check_missing_installations.sh index dd292e4..5ea7c5a 100755 --- a/check_missing_installations.sh +++ b/check_missing_installations.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Script to check for missing installations in NESSI software stack (version 2023.06) +# Script to check for missing installations in EESSI software stack # # author: Kenneth Hoste (@boegel) # author: Thomas Roeblitz (@trz42) ```configure_easybuild- [ ]diff
```diff diff --git a/configure_easybuild b/configure_easybuild index c1bd1d3..c67b879 100644 --- a/configure_easybuild +++ b/configure_easybuild @@ -25,7 +25,6 @@ fi # note: filtering Bison may break some installations, like Qt5 (see https://github.com/EESSI/software-layer/issues/49) # filtering pkg-config breaks R-bundle-Bioconductor installation (see also https://github.com/easybuilders/easybuild-easyconfigs/pull/11104) -# problems occur when filtering pkg-config with gnuplot too (picks up Lua 5.1 from $EPREFIX rather than from Lua 5.3 dependency) DEPS_TO_FILTER=Autoconf,Automake,Autotools,binutils,bzip2,DBus,flex,gettext,gperf,help2man,intltool,libreadline,libtool,M4,makeinfo,ncurses,util-linux,XZ,zlib # For aarch64 we need to also filter out Yasm. # See https://github.com/easybuilders/easybuild-easyconfigs/issues/11190 ```create_directory_tarballs.sh- [ ]diff
```diff diff --git a/create_directory_tarballs.sh b/create_directory_tarballs.sh index a1f05b5..0270719 100755 --- a/create_directory_tarballs.sh +++ b/create_directory_tarballs.sh @@ -9,7 +9,7 @@ fi version=$1 -SOFTWARE_LAYER_TARBALL_URL=https://github.com/NorESSI/software-layer/tarball/nessi.no-2023.06 +SOFTWARE_LAYER_TARBALL_URL="https://github.com/EESSI/software-layer/tarball/${version}-software.eessi.io" TOPDIR=$(dirname $(realpath $0)) ```create_tarball.sh- [ ]diff
```diff diff --git a/create_tarball.sh b/create_tarball.sh index f6239cf..a619df9 100755 --- a/create_tarball.sh +++ b/create_tarball.sh @@ -43,7 +43,7 @@ module_files_list=${tmpdir}/module_files.list.txt if [ -d ${eessi_version}/software/${os}/${cpu_arch_subdir}/.lmod ]; then # include Lmod cache and configuration file (lmodrc.lua), # skip whiteout files and backup copies of Lmod cache (spiderT.old.*) - find ${eessi_version}/software/${os}/${cpu_arch_subdir}/.lmod -type f | egrep -v '/\.wh\.|spiderT.old' > ${files_list} + find ${eessi_version}/software/${os}/${cpu_arch_subdir}/.lmod -type f | egrep -v '/\.wh\.|spiderT.old' >> ${files_list} fi # include scripts that were copied by install_scripts.sh, which we want to ship in EESSI repository ```easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml- [ ]diff
```diff diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml deleted file mode 100644 index 28612c5..0000000 --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml +++ /dev/null @@ -1,27 +0,0 @@ -easyconfigs: - - OpenSSL-1.1.eb - # - OpenSSL-1.1.eb: - # options: - # include-easyblocks-from-pr: 2922 - - git-2.32.0-GCCcore-10.3.0-nodocs.eb - - GCC-10.3.0 - - CMake-3.20.1-GCCcore-10.3.0.eb - # - CMake-3.20.1-GCCcore-10.3.0.eb: - # options: - # include-easyblocks-from-pr: 2248 - - Rust-1.52.1-GCCcore-10.3.0.eb - - foss-2021a.eb - - QuantumESPRESSO-6.7-foss-2021a.eb - - GROMACS-2021.3-foss-2021a.eb: - options: - download-timeout: 1000 - - libGLU-9.0.1-GCCcore-10.3.0.eb - # since the BAGEL ec file is new we have to keep the from-pr or eb fails - - BAGEL-1.2.2-foss-2021a.eb: - options: - from-pr: 18446 - - WRF-4.3-foss-2021a-dmpar.eb - - Yambo-5.1.1-foss-2021a.eb: - options: - from-pr: 18905 - ```easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml- [ ]diff
```diff diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml deleted file mode 100644 index d838e15..0000000 --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml +++ /dev/null @@ -1,18 +0,0 @@ -easyconfigs: - - GCC-11.2.0 - - git-2.33.1-GCCcore-11.2.0-nodocs.eb - - CMake-3.21.1-GCCcore-11.2.0.eb - # - CMake-3.21.1-GCCcore-11.2.0.eb: - # options: - # include-easyblocks-from-pr: 2248 - - OpenMPI-4.1.1-GCC-11.2.0.eb - - FFTW-3.3.10-gompi-2021b.eb - - BLIS-0.8.1-GCC-11.2.0.eb - - foss-2021b.eb - # note, the below CMake has never been installed, just put here to use the - # same version as easy does - # - CMake-3.22.1-GCCcore-11.2.0.eb: - # options: - # include-easyblocks-from-pr: 2248 - # - zlib-1.2.11-GCCcore-11.2.0 - - QuantumESPRESSO-6.8-foss-2021b.eb ```easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml- [ ]diff
```diff diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml deleted file mode 100644 index 9991de1..0000000 --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml +++ /dev/null @@ -1,12 +0,0 @@ -easyconfigs: - - GCC-11.3.0 - - git-2.36.0-GCCcore-11.3.0-nodocs.eb - - CMake-3.23.1-GCCcore-11.3.0.eb - # - CMake-3.23.1-GCCcore-11.3.0.eb: - # options: - # include-easyblocks-from-pr: 2248 - - BLIS-0.9.0-GCC-11.3.0.eb - - OpenMPI-4.1.4-GCC-11.3.0.eb - - FFTW.MPI-3.3.10-gompi-2022a.eb - - foss-2022a.eb - - Python-3.10.4-GCCcore-11.3.0.eb ```easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml- [ ]diff
```diff diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml deleted file mode 100644 index 129b709..0000000 --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml +++ /dev/null @@ -1,12 +0,0 @@ -easyconfigs: - - GCC-12.2.0 - - git-2.38.1-GCCcore-12.2.0-nodocs.eb - - CMake-3.24.3-GCCcore-12.2.0.eb - # - CMake-3.24.3-GCCcore-12.2.0.eb: - # options: - # include-easyblocks-from-pr: 2248 - - BLIS-0.9.0-GCC-12.2.0.eb - - OpenMPI-4.1.4-GCC-12.2.0.eb - - FFTW.MPI-3.3.10-gompi-2022b.eb - - Python-3.10.8-GCCcore-12.2.0.eb - - foss-2022b.eb ```easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml- [ ]diff
```diff diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml deleted file mode 100644 index a5f84c7..0000000 --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml +++ /dev/null @@ -1,4 +0,0 @@ -easyconfigs: - - EasyBuild-4.8.0.eb: - options: - from-pr: 18282 ```easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml- [ ]diff
```diff diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml deleted file mode 100644 index 96d79de..0000000 --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml +++ /dev/null @@ -1,4 +0,0 @@ -easyconfigs: - - EasyBuild-4.8.1.eb: - options: - from-pr: 18761 ```easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml- [ ]diff
```diff diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml deleted file mode 100644 index 14f936b..0000000 --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml +++ /dev/null @@ -1,66 +0,0 @@ -easyconfigs: - - BAMM-2.5.0-foss-2022a.eb - - BamTools-2.5.2-GCC-11.3.0.eb - - BCFtools-1.15.1-GCC-11.3.0.eb - - Bowtie2-2.4.5-GCC-11.3.0.eb - - FastTree-2.1.11-GCCcore-11.3.0.eb - - SAMtools-1.16.1-GCC-11.3.0.eb - - gzip-1.12-GCCcore-11.3.0.eb - - lz4-1.9.3-GCCcore-11.3.0.eb - - zstd-1.5.2-GCCcore-11.3.0.eb - - Pillow-9.1.1-GCCcore-11.3.0.eb: - # avoid that hardcoded paths like /usr/include are used in build commands - options: - from-pr: 18881 - - Boost.Python-1.79.0-GCC-11.3.0.eb - - netCDF-4.9.0-gompi-2022a.eb: - # use updated CMakeMake easyblock to avoid that -DCMAKE_SKIP_RPATH=ON is used, which breaks the netCDF test step - # see https://github.com/easybuilders/easybuild-easyblocks/pull/3012 - options: - include-easyblocks-from-pr: 3012 - - NCO-5.1.0-foss-2022a.eb - - AdapterRemoval-2.3.3-GCC-11.3.0.eb - - BEDTools-2.30.0-GCC-11.3.0.eb - - Pysam-0.20.0-GCC-11.3.0.eb - - TransDecoder-5.5.0-GCC-11.3.0.eb - - VCFtools-0.1.16-GCC-11.3.0.eb - - XML-LibXML-2.0207-GCCcore-11.3.0.eb - - CMake-3.24.3-GCCcore-11.3.0.eb - - elfutils-0.187-GCCcore-11.3.0.eb - - Ninja-1.10.2-GCCcore-11.3.0.eb - - Z3-4.10.2-GCCcore-11.3.0.eb - - SciPy-bundle-2022.05-foss-2022a.eb - - Xvfb-21.1.3-GCCcore-11.3.0.eb: - # enable exec permissions for xvfb-run after copying; - # need to also enable user write permissions on xvfb-run to ensure that copying with preserved permissions works - options: - from-pr: 18834 - - tbb-2021.5.0-GCCcore-11.3.0.eb - - CMSeq-1.0.4-foss-2022a.eb - - Pango-1.50.7-GCCcore-11.3.0.eb - - nodejs-16.15.1-GCCcore-11.3.0.eb: - options: - download-timeout: 1000 - - ecCodes-2.27.0-gompi-2022a.eb: - options: - download-timeout: 1000 - - GLPK-5.0-GCCcore-11.3.0.eb - - libgit2-1.4.3-GCCcore-11.3.0.eb - - libGLU-9.0.2-GCCcore-11.3.0.eb - - libsndfile-1.1.0-GCCcore-11.3.0.eb - - LibTIFF-4.3.0-GCCcore-11.3.0.eb - - MPFR-4.1.0-GCCcore-11.3.0.eb - - NLopt-2.7.1-GCCcore-11.3.0.eb - - PCRE2-10.40-GCCcore-11.3.0.eb - - Tk-8.6.12-GCCcore-11.3.0.eb - - GROMACS-2023.1-foss-2022a.eb - - MUMPS-5.5.1-foss-2022a-metis.eb - - Pillow-9.1.1-GCCcore-11.3.0.eb: - # Uses a custom hook since has zlib as dependency which has hard coded header and library path within Pillow code. - options: - from-pr: 18881 - - ParaView-5.10.1-foss-2022a-mpi.eb: - options: - download-timeout: 1000 - - ASE-3.22.1-foss-2022a.eb - - HISAT2-2.2.1-gompi-2022a.eb ```easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml- [ ]diff
```diff diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml deleted file mode 100644 index f191e76..0000000 --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml +++ /dev/null @@ -1,3 +0,0 @@ -easyconfigs: - - CDO-2.0.6-gompi-2022a.eb - - OpenBabel-3.1.1-gompi-2022a.eb ```easystacks/pilot.nessi.no/2023.06/README.md- [ ]diff
```diff diff --git a/easystacks/pilot.nessi.no/2023.06/README.md b/easystacks/software.eessi.io/2023.06/README.md similarity index 100% rename from easystacks/pilot.nessi.no/2023.06/README.md rename to easystacks/software.eessi.io/2023.06/README.md ```easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml- [ ]diff
```diff diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-001-system.yml similarity index 54% rename from easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml rename to easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-001-system.yml index 7418c0a..f02b9f2 100644 --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-001-system.yml @@ -1,6 +1,7 @@ easyconfigs: - # wraps around Java/11.0.20 - - Java-11.eb - EasyBuild-4.8.2.eb: options: from-pr: 19105 + - Nextflow-23.10.0.eb: + options: + from-pr: 19172 ```easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml- [ ]diff
```diff diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml new file mode 100644 index 0000000..dda274b --- /dev/null +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml @@ -0,0 +1,5 @@ +easyconfigs: + - CUDA-12.1.1.eb: + options: + include-easyblocks-from-pr: 3045 + accept-eula-for: CUDA ```easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml- [ ]diff
```diff diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml new file mode 100644 index 0000000..69ffb75 --- /dev/null +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml @@ -0,0 +1,6 @@ +easyconfigs: + - foss-2022b.eb + - HarfBuzz-5.3.1-GCCcore-12.2.0.eb: + options: + from-pr: 19339 + - Qt5-5.15.7-GCCcore-12.2.0.eb ```easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml- [ ]diff
```diff diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml new file mode 100644 index 0000000..aab910d --- /dev/null +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml @@ -0,0 +1,48 @@ +easyconfigs: + - GCC-12.3.0.eb + - Rust-1.70.0-GCCcore-12.3.0.eb: + # fix build of Rust 1.70.0 by disabling download of pre-built LLVM; + # see https://github.com/easybuilders/easybuild-easyblocks/pull/3038 + options: + include-easyblocks-from-pr: 3038 + - foss-2023a.eb + - pybind11-2.11.1-GCCcore-12.3.0.eb: + # avoid indirect dependency on old CMake version built with GCCcore/10.2.0 via Catch2 build dependency; + # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19270 + options: + from-pr: 19270 + - SciPy-bundle-2023.07-gfbf-2023a.eb + - TensorFlow-2.13.0-foss-2023a.eb: + # patch setup.py for grpcio extension in TensorFlow 2.13.0 easyconfigs to take into account alternate sysroot; + # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19268 + options: + from-pr: 19268 + - X11-20230603-GCCcore-12.3.0.eb + - HarfBuzz-5.3.1-GCCcore-12.3.0.eb: + options: + from-pr: 19339 + - Qt5-5.15.10-GCCcore-12.3.0.eb + - OSU-Micro-Benchmarks-7.1-1-gompi-2023a.eb + - LHAPDF-6.5.4-GCC-12.3.0.eb: + options: + from-pr: 19363 + - LoopTools-2.15-GCC-12.3.0.eb: + options: + from-pr: 19397 + - R-4.3.2-gfbf-2023a.eb: + options: + from-pr: 19185 + - Boost-1.82.0-GCC-12.3.0.eb + - netCDF-4.9.2-gompi-2023a.eb + - FFmpeg-6.0-GCCcore-12.3.0.eb + - CUDA-Samples-12.1-GCC-12.3.0-CUDA-12.1.1.eb: + # use easyconfig that only install subset of CUDA samples, + # to circumvent problem with nvcc linking to glibc of host OS, + # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19189; + # and where additional samples are excluded because they fail to build on aarch64, + # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19451; + options: + from-pr: 19451 + - ALL-0.9.2-foss-2023a.eb: + options: + from-pr: 19455 ```easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml- [ ]diff
```diff diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml new file mode 100644 index 0000000..25c13e4 --- /dev/null +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml @@ -0,0 +1,5 @@ +easyconfigs: + - EasyBuild-4.9.0.eb: + options: + from-pr: 19464 + - ReFrame-4.3.3.eb ```easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml- [ ]diff
```diff diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml new file mode 100644 index 0000000..fdacd95 --- /dev/null +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml @@ -0,0 +1,2 @@ +easyconfigs: + - SciPy-bundle-2023.02-gfbf-2022b.eb ```easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml- [ ]diff
```diff diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml new file mode 100644 index 0000000..eba776a --- /dev/null +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml @@ -0,0 +1,10 @@ +easyconfigs: + - OpenFOAM-11-foss-2023a.eb: + # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19545 + options: + from-pr: 19545 + - at-spi2-core-2.49.91-GCCcore-12.3.0.eb + - ESPResSo-4.2.1-foss-2023a.eb: + # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19592 + options: + from-pr: 19592 ```easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml- [ ]diff
```diff diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml new file mode 100644 index 0000000..9a3b8ef --- /dev/null +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml @@ -0,0 +1,7 @@ +easyconfigs: + - GCC-13.2.0.eb + - foss-2023b.eb + - SciPy-bundle-2023.11-gfbf-2023b.eb + - netCDF-4.9.2-gompi-2023b.eb: + options: + from-pr: 19534 ```eb_hooks.py- [ ]diff
```diff diff --git a/eb_hooks.py b/eb_hooks.py index b3cbc4b..2bc2fde 100644 --- a/eb_hooks.py +++ b/eb_hooks.py @@ -185,20 +185,21 @@ def parse_hook_fontconfig_add_fonts(ec, eprefix): def parse_hook_openblas_relax_lapack_tests_num_errors(ec, eprefix): - """Relax number of failing numerical LAPACK tests for aarch64/neoverse_v1 CPU target.""" + """Relax number of failing numerical LAPACK tests for aarch64/neoverse_v1 CPU target for OpenBLAS < 0.3.23""" cpu_target = get_eessi_envvar('EESSI_SOFTWARE_SUBDIR') if ec.name == 'OpenBLAS': - # relax maximum number of failed numerical LAPACK tests for aarch64/neoverse_v1 CPU target - # since the default setting of 150 that works well on other aarch64 targets and x86_64 is a bit too strict - # See https://github.com/EESSI/software-layer/issues/314 - cfg_option = 'max_failing_lapack_tests_num_errors' - if cpu_target == CPU_TARGET_NEOVERSE_V1: - orig_value = ec[cfg_option] - ec[cfg_option] = 400 - print_msg("Maximum number of failing LAPACK tests with numerical errors for %s relaxed to %s (was %s)", - ec.name, ec[cfg_option], orig_value) - else: - print_msg("Not changing option %s for %s on non-AARCH64", cfg_option, ec.name) + if LooseVersion(ec.version) < LooseVersion('0.3.23'): + # relax maximum number of failed numerical LAPACK tests for aarch64/neoverse_v1 CPU target + # since the default setting of 150 that works well on other aarch64 targets and x86_64 is a bit too strict + # See https://github.com/EESSI/software-layer/issues/314 + cfg_option = 'max_failing_lapack_tests_num_errors' + if cpu_target == CPU_TARGET_NEOVERSE_V1: + orig_value = ec[cfg_option] + ec[cfg_option] = 400 + print_msg("Maximum number of failing LAPACK tests with numerical errors for %s relaxed to %s (was %s)", + ec.name, ec[cfg_option], orig_value) + else: + print_msg("Not changing option %s for %s on non-AARCH64", cfg_option, ec.name) else: raise EasyBuildError("OpenBLAS-specific hook triggered for non-OpenBLAS easyconfig?!") @@ -393,6 +394,18 @@ def pre_test_hook_ignore_failing_tests_SciPybundle(self, *args, **kwargs): if self.name == 'SciPy-bundle' and self.version in scipy_bundle_versions and cpu_target == CPU_TARGET_NEOVERSE_V1: self.cfg['testopts'] = "|| echo ignoring failing tests" +def pre_test_hook_ignore_failing_tests_netCDF(self, *args, **kwargs): + """ + Pre-test hook for netCDF: skip failing tests for selected netCDF versions on neoverse_v1 + cfr. https://github.com/EESSI/software-layer/issues/425 + The following tests are problematic: + 163 - nc_test4_run_par_test (Timeout) + 190 - h5_test_run_par_tests (Timeout) + A few other tests are skipped in the easyconfig and patches for similar issues, see above issue for details. + """ + cpu_target = get_eessi_envvar('EESSI_SOFTWARE_SUBDIR') + if self.name == 'netCDF' and self.version == '4.9.2' and cpu_target == CPU_TARGET_NEOVERSE_V1: + self.cfg['testopts'] = "|| echo ignoring failing tests" def pre_single_extension_hook(ext, *args, **kwargs): """Main pre-extension: trigger custom functions based on software name.""" @@ -573,6 +586,7 @@ PRE_TEST_HOOKS = { 'ESPResSo': pre_test_hook_ignore_failing_tests_ESPResSo, 'FFTW.MPI': pre_test_hook_ignore_failing_tests_FFTWMPI, 'SciPy-bundle': pre_test_hook_ignore_failing_tests_SciPybundle, + 'netCDF': pre_test_hook_ignore_failing_tests_netCDF, } PRE_SINGLE_EXTENSION_HOOKS = { ```eessi-2023.06-known-issues.yml- [ ]diff
```diff diff --git a/eessi-2023.06-known-issues.yml b/eessi-2023.06-known-issues.yml new file mode 100644 index 0000000..475ee2c --- /dev/null +++ b/eessi-2023.06-known-issues.yml @@ -0,0 +1,28 @@ +- aarch64/neoverse_v1: + - ESPResSo-4.2.1-foss-2023a: + - issue: https://github.com/EESSI/software-layer/issues/363 + - info: "ESPResSo tests failing due to timeouts" + - FFTW.MPI-3.3.10-gompi-2023a: + - issue: https://github.com/EESSI/software-layer/issues/325 + - info: "Flaky FFTW tests, random failures" + - FFTW.MPI-3.3.10-gompi-2023b: + - issue: https://github.com/EESSI/software-layer/issues/325 + - info: "Flaky FFTW tests, random failures" + - netCDF-4.9.2-gompi-2023a.eb: + - issue: https://github.com/EESSI/software-layer/issues/425 + - info: "netCDF intermittent test failures" + - netCDF-4.9.2-gompi-2023b.eb: + - issue: https://github.com/EESSI/software-layer/issues/425 + - info: "netCDF intermittent test failures" + - OpenBLAS-0.3.21-GCC-12.2.0: + - issue: https://github.com/EESSI/software-layer/issues/314 + - info: "Increased number of numerical errors in OpenBLAS test suite (344 vs max. 150 on x86_64/*)" + - SciPy-bundle-2023.02-gfbf-2022b: + - issue: https://github.com/EESSI/software-layer/issues/318 + - info: "numpy built with -march=armv8.4-a instead of -mcpu=native (no SVE) + 2 failing tests (vs 50005 passed) in scipy test suite" + - SciPy-bundle-2023.07-gfbf-2023a: + - issue: https://github.com/EESSI/software-layer/issues/318 + - info: "2 failing tests (vs 54409 passed) in scipy test suite" + - SciPy-bundle-2023.11-gfbf-2023b: + - issue: https://github.com/EESSI/software-layer/issues/318 + - info: "2 failing tests (vs 54876 passed) in scipy test suite" ```eessi_container.sh- [ ]diff
```diff diff --git a/eessi_container.sh b/eessi_container.sh index bfc5963..d6e9558 100755 --- a/eessi_container.sh +++ b/eessi_container.sh @@ -30,8 +30,8 @@ # -. initial settings & exit codes TOPDIR=$(dirname $(realpath $0)) -source ${TOPDIR}/scripts/utils.sh -source ${TOPDIR}/scripts/cfg_files.sh +source "${TOPDIR}"/scripts/utils.sh +source "${TOPDIR}"/scripts/cfg_files.sh # exit codes: bitwise shift codes to allow for combination of exit codes # ANY_ERROR_EXITCODE is sourced from ${TOPDIR}/scripts/utils.sh @@ -83,7 +83,7 @@ display_help() { echo " MODE==run (run a script or command) [default: shell]" echo " -n | --nvidia MODE - configure the container to work with NVIDIA GPUs," echo " MODE==install for a CUDA installation, MODE==run to" - echo " attach a GPU, MODE==all for both [default: false]" + echo " attach a GPU, MODE==all for both [default: false]" echo " -r | --repository CFG - configuration file or identifier defining the" echo " repository to use [default: EESSI via" echo " default container, see --container]" @@ -164,7 +164,7 @@ while [[ $# -gt 0 ]]; do SETUP_NVIDIA=1 NVIDIA_MODE="$2" shift 2 - ;; + ;; -r|--repository) REPOSITORY="$2" shift 2 @@ -575,7 +575,7 @@ fi declare -a EESSI_FUSE_MOUNTS=() # always mount cvmfs-config repo (to get access to software.eessi.io) -# EESSI_FUSE_MOUNTS+=("--fusemount" "container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch") +EESSI_FUSE_MOUNTS+=("--fusemount" "container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch") if [[ "${ACCESS}" == "ro" ]]; then export EESSI_READONLY="container:cvmfs2 ${repo_name} /cvmfs/${repo_name}" ```init/Magic_Castle/bash- [ ]diff
```diff diff --git a/init/Magic_Castle/bash b/init/Magic_Castle/bash index 0fb2f67..dbe1772 100644 --- a/init/Magic_Castle/bash +++ b/init/Magic_Castle/bash @@ -6,7 +6,7 @@ EESSI_SILENT=1 source $(dirname "$BASH_SOURCE")/../eessi_environment_variables # Don't change the default prompt -# export PS1="{NESSI $EESSI_VERSION} $ " +# export PS1="[EESSI $EESSI_VERSION] $ " # Provide a clean MODULEPATH export MODULEPATH_ROOT=$EESSI_MODULEPATH @@ -36,4 +36,4 @@ else module reload fi -echo "Environment set up to use NESSI (${EESSI_VERSION}), have fun!" +echo "Environment set up to use EESSI (${EESSI_VERSION}), have fun!" ```init/Magic_Castle/eessi_python3- [ ]diff
```diff diff --git a/init/Magic_Castle/eessi_python3 b/init/Magic_Castle/eessi_python3 index c76c7a2..b2f7fd4 100755 --- a/init/Magic_Castle/eessi_python3 +++ b/init/Magic_Castle/eessi_python3 @@ -16,6 +16,6 @@ eessi_python=$(ls ${EESSI_SOFTWARE_PATH}/software/Python/3*GCCcore*/bin/python | if [ -f "$eessi_python" ]; then $eessi_python "$@" else - echo "ERROR: No NESSI Python 3 available." + echo "ERROR: No EESSI Python 3 available." false fi ```init/README.md- [ ]diff
```diff diff --git a/init/README.md b/init/README.md index 05baffd..c253dc4 100644 --- a/init/README.md +++ b/init/README.md @@ -1,7 +1,7 @@ -# Bash initialisation for NESSI +# Bash initialisation for EESSI This directory contains the default initialisation script for a bash shell used to -configure Lmod and use the NESSI software modules. The (bash) +configure Lmod and use the EESSI software modules. The (bash) file `eessi_environment_variables` is used to set and export the full set of EESSI environment variables: @@ -9,12 +9,12 @@ environment variables: - `EESSI_EPREFIX`: The location of Gentoo Prefix compatability layer (for the architecture). - `EESSI_EPREFIX_PYTHON`: Path to `python3` in the Gentoo Prefix layer. - `EESSI_SOFTWARE_SUBDIR`: Hardware specific software subdirectory. -- `EESSI_SOFTWARE_PATH`: Full path to NESSI software stack. +- `EESSI_SOFTWARE_PATH`: Full path to EESSI software stack. - `EESSI_MODULEPATH`: Path to be added to the `MODULEPATH`. This can be influenced by two externally defined environment varialbes: - `EESSI_CUSTOM_MODULEPATH`: defines a fully custom directory to be added to `MODULEPATH`, the end user is entirely responsible for what this directory contains. - - `EESSI_MODULE_SUBDIR`: NESSI may ship with a number of possible module naming schemes. + - `EESSI_MODULE_SUBDIR`: EESSI may ship with a number of possible module naming schemes. This variable can be used to point to a non-default module naming scheme. All scripts respect the environment variable `EESSI_SILENT` which, if defined to any @@ -25,5 +25,5 @@ value, will make them produce no (non-error) output. The `Magic_Castle` subdirectory is home to the bash initialisation that we use for [Magic Castle](https://github.com/ComputeCanada/magic_castle). -It also contains a wrapper for a hardware optimised NESSI Python 3 installation that is used +It also contains a wrapper for a hardware optimised EESSI Python 3 installation that is used by Magic Castle to properly configure JupyterHub. ```init/arch_specs/eessi_arch_arm.spec- [ ]diff
```diff diff --git a/init/arch_specs/eessi_arch_arm.spec b/init/arch_specs/eessi_arch_arm.spec index f8f21f9..b5c9275 100755 --- a/init/arch_specs/eessi_arch_arm.spec +++ b/init/arch_specs/eessi_arch_arm.spec @@ -1,6 +1,6 @@ # ARM CPU architecture specifications # Software path in EESSI | Vendor ID | List of defining CPU features -"aarch64/neoverse-n1" "ARM" "asimd" # Ampere Altra -"aarch64/neoverse-n1" "" "asimd" # AWS Graviton2 -"aarch64/neoverse-v1" "ARM" "asimd svei8mm" -"aarch64/neoverse-v1" "" "asimd svei8mm" # AWS Graviton3 +"aarch64/neoverse_n1" "ARM" "asimd" # Ampere Altra +"aarch64/neoverse_n1" "" "asimd" # AWS Graviton2 +"aarch64/neoverse_v1" "ARM" "asimd svei8mm" +"aarch64/neoverse_v1" "" "asimd svei8mm" # AWS Graviton3 ```init/bash- [ ]diff
```diff diff --git a/init/bash b/init/bash index 26598bb..0029f91 100644 --- a/init/bash +++ b/init/bash @@ -5,6 +5,7 @@ function show_msg { echo "$msg" fi } + # The following method should be safe, but might break if file is a symlink # (could switch to $(dirname "$(readlink -f "$BASH_SOURCE")") in that case) source $(dirname "$BASH_SOURCE")/eessi_environment_variables @@ -12,7 +13,7 @@ source $(dirname "$BASH_SOURCE")/eessi_environment_variables # only continue if setting EESSI environment variables worked fine if [ $? -eq 0 ]; then - export PS1="{NESSI $EESSI_VERSION} $PS1" + export PS1="{EESSI $EESSI_VERSION} $PS1" # add location of commands provided by compat layer to $PATH; # see https://github.com/EESSI/software-layer/issues/52 @@ -23,7 +24,7 @@ if [ $? -eq 0 ]; then source $EESSI_EPREFIX/usr/share/Lmod/init/bash # prepend location of modules for EESSI software stack to $MODULEPATH - echo "Prepending $EESSI_MODULEPATH to \$MODULEPATH..." >> $output + show_msg "Prepending $EESSI_MODULEPATH to \$MODULEPATH..." module use $EESSI_MODULEPATH #show_msg "" @@ -33,6 +34,6 @@ if [ $? -eq 0 ]; then #show_msg "" #show_msg "" - echo "Environment set up to use NESSI (${EESSI_VERSION}), have fun!" + echo "Environment set up to use EESSI (${EESSI_VERSION}), have fun!" fi ```init/eessi_archdetect.sh- [ ]diff
```diff diff --git a/init/eessi_archdetect.sh b/init/eessi_archdetect.sh index 236ac5a..58b79b0 100755 --- a/init/eessi_archdetect.sh +++ b/init/eessi_archdetect.sh @@ -73,7 +73,7 @@ cpupath(){ [ $EESSI_SOFTWARE_SUBDIR_OVERRIDE ] && echo ${EESSI_SOFTWARE_SUBDIR_OVERRIDE} && exit # Identify the best matching CPU architecture from a list of supported specifications for the host CPU - # Return the path to the installation files in NESSI of the best matching architecture + # Return the path to the installation files in EESSI of the best matching architecture local cpu_arch_spec=() # Identify the host CPU architecture @@ -118,7 +118,7 @@ cpupath(){ # each flag in this CPU specification must be found in the list of flags of the host check_allinfirst "${cpu_flags[*]}" ${arch_spec[2]} && best_arch_match=${arch_spec[0]} && \ all_arch_matches="$best_arch_match:$all_arch_matches" && \ - log "DEBUG" "cpupath: host CPU best match updated to $best_arch_match" + log "DEBUG" "cpupath: host CPU best match updated to $best_arch_match" fi done ```init/eessi_defaults- [ ]diff
```diff diff --git a/init/eessi_defaults b/init/eessi_defaults index da057fb..d1779a3 100644 --- a/init/eessi_defaults +++ b/init/eessi_defaults @@ -8,9 +8,8 @@ # license: GPLv2 # -export EESSI_CVMFS_REPO="${EESSI_CVMFS_REPO_OVERRIDE:=/cvmfs/pilot.nessi.no}" +export EESSI_CVMFS_REPO="${EESSI_CVMFS_REPO_OVERRIDE:=/cvmfs/software.eessi.io}" export EESSI_VERSION="${EESSI_VERSION_OVERRIDE:=2023.06}" # use archdetect by default, unless otherwise specified -# 2024-01-16: switch back to archspec with NESSI for now -export EESSI_USE_ARCHDETECT="${EESSI_USE_ARCHDETECT:=0}" -export EESSI_USE_ARCHSPEC="${EESSI_USE_ARCHSPEC:=1}" +export EESSI_USE_ARCHDETECT="${EESSI_USE_ARCHDETECT:=1}" +export EESSI_USE_ARCHSPEC="${EESSI_USE_ARCHSPEC:=0}" ```init/eessi_environment_variables- [ ]diff
```diff diff --git a/init/eessi_environment_variables b/init/eessi_environment_variables index d4a2e72..3584bfa 100644 --- a/init/eessi_environment_variables +++ b/init/eessi_environment_variables @@ -2,6 +2,11 @@ # $BASH_SOURCE points to correct path, see also http://mywiki.wooledge.org/BashFAQ/028 EESSI_INIT_DIR_PATH=$(dirname $(realpath $BASH_SOURCE)) +function error() { + echo -e "\e[31mERROR: $1\e[0m" >&2 + false +} + function show_msg { # only echo msg if EESSI_SILENT is unset msg=$1 @@ -10,16 +15,11 @@ function show_msg { fi } -function error() { - echo -e "\e[31mERROR: $1\e[0m" >&2 - false -} - # set up minimal environment: $EESSI_PREFIX, $EESSI_VERSION, $EESSI_OS_TYPE, $EESSI_CPU_FAMILY, $EPREFIX source $EESSI_INIT_DIR_PATH/minimal_eessi_env if [ -d $EESSI_PREFIX ]; then - show_msg "Found NESSI repo @ $EESSI_PREFIX!" + show_msg "Found EESSI repo @ $EESSI_PREFIX!" export EESSI_EPREFIX=$EPREFIX if [ -d $EESSI_EPREFIX ]; then @@ -63,7 +63,7 @@ if [ -d $EESSI_PREFIX ]; then export EESSI_MODULEPATH=$EESSI_MODULEPATH show_msg "Using ${EESSI_MODULEPATH} as the directory to be added to MODULEPATH." else - error "NESSI module path at $EESSI_MODULEPATH not found!" + error "EESSI module path at $EESSI_MODULEPATH not found!" false fi @@ -75,7 +75,7 @@ if [ -d $EESSI_PREFIX ]; then fi else - error "NESSI software layer at $EESSI_SOFTWARE_PATH not found!" + error "EESSI software layer at $EESSI_SOFTWARE_PATH not found!" fi else error "no value set for \$EESSI_SOFTWARE_SUBDIR" @@ -84,5 +84,5 @@ if [ -d $EESSI_PREFIX ]; then error "Compatibility layer directory $EESSI_EPREFIX not found!" fi else - error "NESSI repository at $EESSI_PREFIX not found!" + error "EESSI repository at $EESSI_PREFIX not found!" fi ```init/eessi_software_subdir_for_host.py- [ ]diff
```diff diff --git a/init/eessi_software_subdir_for_host.py b/init/eessi_software_subdir_for_host.py index e8f1a71..58e9cfd 100755 --- a/init/eessi_software_subdir_for_host.py +++ b/init/eessi_software_subdir_for_host.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 # -# Determine NESSI software subdirectory to use for current build host, using archspec +# Determine EESSI software subdirectory to use for current build host, using archspec # import glob import os ```init/minimal_eessi_envdiff
```diff diff --git a/init/minimal_eessi_env b/init/minimal_eessi_env index f938d1d..5b35620 100644 --- a/init/minimal_eessi_env +++ b/init/minimal_eessi_env @@ -1,4 +1,4 @@ -# define minimal NESSI environment, without relying on external scripts +# define minimal EESSI environment, without relying on external scripts # # this script is *sourced*, not executed, so can't rely on $0 to determine path to self # $BASH_SOURCE points to correct path, see also http://mywiki.wooledge.org/BashFAQ/028 ``` -
sync
scriptsandinitdirectories of/cvmfs/pilot.nessi.nowith changes in/cvmfs/software.eessi.io- CVMFS repo
pilot.nessi.nosynced with EESSI: - removed symlink
/cvmfs/pilot.nessi.no/latest - added
/cvmfs/pilot.nessi.no/README.nessi - synced/populated
/cvmfs/pilot.nessi.no/versions/2023.06/{init,scripts} - should be available soon on CVMFS clients
- CVMFS repo
-
check Broadwell supported via archdetect (for building packages)
- building works without changes
- using requires an update to a spec file, see #248
-
check archdetect is used correctly via
source /cvmfs/.../init/bash(for using packages)- works for Broadwell
[thomarob@login-2.FRAM ~]$ source /cvmfs/pilot.nessi.no/versions/2023.06/init/bash Found NESSI repo @ /cvmfs/pilot.nessi.no/versions/2023.06! archdetect says x86_64/intel/broadwell Using x86_64/intel/broadwell as software subdirectory. Using /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/modules/all as the directory to be added to MODULEPATH. Found Lmod configuration file at /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/.lmod/lmodrc.lua Initializing Lmod... Prepending /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/modules/all to $MODULEPATH... Environment set up to use NESSI (2023.06), have fun!
- works for Broadwell
-
fix failing CI workflows
TopRichard
NESSI and EESSI have diverted, so in the attempt of syncing them back as close as possible, the following PRs have been merged:
NOTE: CUDA/NVIDIA MODIFICATIONS ARE NOT TESTED!!!
217 --> reverted back, the changes were many and the outcome wasn't as expected
218
220
221
222
224
226
225
227
228
229
231
232
234
235
236
237
238
239
240
241
Test PRs
216
219
230 :+1:
233 :+1:
For now (EESSI Merged PR#454) there exist differences within 74 files as listed below, must carry out resolving those differences carefully:
241
237
238
240
239
221 (directory structure changes)
226
234
227
228 (requires changes of the main job script run by bot instances)
229
220 (removed by this PR)
235
236
236