Open mkandes opened 2 weeks ago
Before creating a new featture branch to deploy Intel TBB, you need to resync the deployment branch of your fork with the sdsc/spack upstream.
[mkandes@login01 spack]$ pwd
/home/mkandes/software/spack/repositories/mkandes/spack
[mkandes@login01 spack]$ git log -1
commit 2eccf0894939f5e1f34d8bde648884b6ad5f0d79 (HEAD -> sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf, origin/sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf)
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Fri Aug 16 11:33:00 2024 -0700
Add nvhpc@21.9 % gcc@8.5.0 + netcdf-* to exp/0.17.3/cpu/b
We are deploying nvhpc@21.9 to exp/0.17.3/cpu/b as a compiler option to
support a specific use case [1], where the end user can currently only
compile their custom code with PGI-based compilers and a set of
compatible NetCDF libraries.
[1] https://github.com/sdsc/spack/issues/125
[mkandes@login01 spack]$ git branch
sdsc-0.17.3
sdsc-0.17.3-gh-124-inst-exp-cpu-dev
* sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf
[mkandes@login01 spack]$ git checkout sdsc-0.17.3
Switched to branch 'sdsc-0.17.3'
Your branch is up to date with 'origin/sdsc-0.17.3'.
[mkandes@login01 spack]$
Resyncing from upstream ...
[mkandes@login01 spack]$ git pull upstream sdsc-0.17.3
From https://github.com/sdsc/spack
* branch sdsc-0.17.3 -> FETCH_HEAD
Updating 3cbfd1ba45..a3029ae4c3
Fast-forward
.../cpu/b/specs/cmake@3.21.4.o33147784.exp-15-56 | 653 ++++++++++++++++++
.../expanse/0.17.3/cpu/b/specs/cmake@3.21.4.sh | 68 ++
.../gcc@10.2.0/fftw@3.3.10-omp.o33150584.exp-15-56 | 662 ++++++++++++++++++
.../cpu/b/specs/gcc@10.2.0/fftw@3.3.10-omp.sh | 5 +-
.../0.17.3/cpu/b/specs/gcc@10.2.0/fftw@3.3.10.sh | 2 +-
.../fftw@3.3.10-omp.o33160030.exp-15-56 | 742 +++++++++++++++++++++
.../gcc@10.2.0/mvapich2@2.3.7/fftw@3.3.10-omp.sh | 81 +++
.../specs/gcc@10.2.0/mvapich2@2.3.7/fftw@3.3.10.sh | 2 +-
.../specs/gcc@10.2.0/openmpi@4.1.3/fftw@3.3.10.sh | 2 +-
.../cpu/b/specs/nvhpc@21.9.o33146549.exp-15-56 | 643 ++++++++++++++++++
.../sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9.sh | 72 ++
.../nvhpc@21.9/hdf5@1.10.7.o33147827.exp-15-56 | 695 +++++++++++++++++++
.../0.17.3/cpu/b/specs/nvhpc@21.9/hdf5@1.10.7.sh | 68 ++
.../nvhpc@21.9/netcdf-c@4.8.1.o33148039.exp-15-56 | 704 +++++++++++++++++++
.../cpu/b/specs/nvhpc@21.9/netcdf-c@4.8.1.sh | 68 ++
.../netcdf-fortran@4.5.3.o33148097.exp-15-56 | 669 +++++++++++++++++++
.../cpu/b/specs/nvhpc@21.9/netcdf-fortran@4.5.3.sh | 67 ++
.../sdsc/expanse/0.17.3/cpu/b/yamls/compilers.yaml | 16 +
18 files changed, 5213 insertions(+), 6 deletions(-)
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/cmake@3.21.4.o33147784.exp-15-56
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/cmake@3.21.4.sh
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/fftw@3.3.10-omp.o33150584.exp-15-56
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/mvapich2@2.3.7/fftw@3.3.10-omp.o33160030.exp-15-56
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/mvapich2@2.3.7/fftw@3.3.10-omp.sh
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9.o33146549.exp-15-56
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9.sh
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/hdf5@1.10.7.o33147827.exp-15-56
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/hdf5@1.10.7.sh
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/netcdf-c@4.8.1.o33148039.exp-15-56
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/netcdf-c@4.8.1.sh
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/netcdf-fortran@4.5.3.o33148097.exp-15-56
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/nvhpc@21.9/netcdf-fortran@4.5.3.sh
[mkandes@login01 spack]$ git log
commit a3029ae4c3bc806522bf34949c3e4ed995e6206b (HEAD -> sdsc-0.17.3, upstream/sdsc-0.17.3)
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Sat Aug 17 07:58:30 2024 -0700
fftw@3.3.10 % gcc@10.2.0 +mpi +openmp failed builds
Unfortauntely, I have not been able to resolve this issue yet. It's a
bit odd that enabling +openmp causes the package build to fail when we
already have successful builds with +mpi ~openmp. We will need to
investigate this further in the future. For the time being, I will
invetigate alternative compiler, MPI, and FFTW options for QE.
commit 5c15c7b17659e27a70d0dacf62449b589f65edca
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Sat Aug 17 07:19:07 2024 -0700
Add fftw@3.3.10 % gcc@10.2.0 +openmp to exp/0.17.3/cpu/b
Adding +openmp-enabled FFTW3 libraries to the production instances in an
attempt to satisfy a requirement for the latest versions of Quantum
Espresso, which appears to require a multithreaded FFTW-compatible
libary. For example, this is the build error I'm currently running into
[mkandes@login01 spack]$ git push
Username for 'https://github.com': ^C
[mkandes@login01 spack]$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com':
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/mkandes/spack.git
3cbfd1ba45..a3029ae4c3 sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login01 spack]$
Create new feature branch for testing spec installation ...
[mkandes@login01 spack]$ git branch
* sdsc-0.17.3
sdsc-0.17.3-gh-124-inst-exp-cpu-dev
sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf
[mkandes@login01 spack]$ git checkout -b sdsc-0.17.3-gh-127-spec-exp-cpu-intel-tbb
Switched to a new branch 'sdsc-0.17.3-gh-127-spec-exp-cpu-intel-tbb'
[mkandes@login01 spack]$ git branch
sdsc-0.17.3
sdsc-0.17.3-gh-124-inst-exp-cpu-dev
sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf
* sdsc-0.17.3-gh-127-spec-exp-cpu-intel-tbb
[mkandes@login01 spack]$
Deploying intel-tbb@2020.3 to expanse/0.17.3/cpu/b ...
[spack_cpu@login01 ~]$ !7463
srun --partition=ind-shared --reservation=root_73 --account=use300 --nodes=1 --nodelist=exp-15-56 --ntasks-per-node=1 --cpus-per-task=16 --mem=32G --time=12:00:00 --pty --wait=0 /bin/bash
[spack_cpu@exp-15-56 ~]$ git log -1
fatal: not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[spack_cpu@exp-15-56 ~]$ cd /cm/shared/apps/spack/0.17.3/cpu/b
[spack_cpu@exp-15-56 b]$ git log -1
commit a3029ae4c3bc806522bf34949c3e4ed995e6206b (HEAD -> sdsc-0.17.3, origin/sdsc-0.17.3, origin/HEAD)
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Sat Aug 17 07:58:30 2024 -0700
fftw@3.3.10 % gcc@10.2.0 +mpi +openmp failed builds
Unfortauntely, I have not been able to resolve this issue yet. It's a
bit odd that enabling +openmp causes the package build to fail when we
already have successful builds with +mpi ~openmp. We will need to
investigate this further in the future. For the time being, I will
invetigate alternative compiler, MPI, and FFTW options for QE.
[spack_cpu@exp-15-56 b]$ git fetch origin
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (6/6), done.
Unpacking objects: 100% (16/16), 3.03 KiB | 119.00 KiB/s, done.
remote: Total 16 (delta 7), reused 15 (delta 7), pack-reused 0 (from 0)
From https://github.com/sdsc/spack
a3029ae4c3..7db33edafe sdsc-0.17.3 -> origin/sdsc-0.17.3
[spack_cpu@exp-15-56 b]$ git log -1
commit a3029ae4c3bc806522bf34949c3e4ed995e6206b (HEAD -> sdsc-0.17.3)
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Sat Aug 17 07:58:30 2024 -0700
fftw@3.3.10 % gcc@10.2.0 +mpi +openmp failed builds
Unfortauntely, I have not been able to resolve this issue yet. It's a
bit odd that enabling +openmp causes the package build to fail when we
already have successful builds with +mpi ~openmp. We will need to
investigate this further in the future. For the time being, I will
invetigate alternative compiler, MPI, and FFTW options for QE.
[spack_cpu@exp-15-56 b]$ git pull
Updating a3029ae4c3..7db33edafe
Fast-forward
activate-shared-spack-instance.sh | 2 +-
configure-shared-spack-instance.sh | 2 +-
.../specs/intel@19.1.3.304/intel-mkl@2020.4.304.sh | 2 +-
.../b/specs/intel@19.1.3.304/intel-tbb@2020.3.sh | 73 ++++++++++++++++++++++
4 files changed, 76 insertions(+), 3 deletions(-)
mode change 100644 => 100755 configure-shared-spack-instance.sh
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/intel@19.1.3.304/intel-tbb@2020.3.sh
[spack_cpu@exp-15-56 b]$ cd etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/
[spack_cpu@exp-15-56 specs]$ cd intel@19.1.3.304/
[spack_cpu@exp-15-56 intel@19.1.3.304]$ sbatch intel-tbb@2020.3.sh
sbatch: error: Batch job submission failed: Requested reservation is invalid
[spack_cpu@exp-15-56 intel@19.1.3.304]$ vi intel-tbb@2020.3.sh
[spack_cpu@exp-15-56 intel@19.1.3.304]$ sbatch intel-tbb@2020.3.sh
Submitted batch job 33452916
[spack_cpu@exp-15-56 intel@19.1.3.304]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
33452916 ind-share intel-tb spack_cp R 0:04 1 exp-15-56
33452904 ind-share bash spack_cp R 1:59 1 exp-15-56
[spack_cpu@exp-15-56 intel@19.1.3.304]$
Deployed successfully ...
[mkandes@login02 ~]$ module load intel/19.1.3.304
[mkandes@login02 ~]$ module avail
/cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/intel/19.1.3.304
arpack-ng/3.8.0/ndrjsnd netcdf-cxx4/4.3.1/d5kubqf
cmake/3.21.4/bpzre3q netcdf-fortran/4.5.3/hg7lhvi
eigen/3.4.0/w2iz4rf netlib-lapack/3.9.1/64ytaps
fftw/3.3.10/jq4mbmk openblas/0.3.18/sctoo2e-omp
gsl/2.7/ix3kgnc openblas/0.3.18/xxq5a3j (D)
hdf5/1.10.7/butdr5c openmpi/4.1.3/luhyajc
hypre/2.23.0/w7eerxg papi/6.0.0.1/tbdvjsm
intel-mkl/2020.4.304/vg6aq26 petsc/3.16.1/ax5r6e3
intel-mpi/2019.10.317/ezrfjne sprng/5.0/u2op57v
intel-tbb/2020.3/lfesfxm superlu/5.3.0/tycq2th
https://access-ci.atlassian.net/browse/ATS-9980