Closed mkandes closed 2 weeks ago
After another review of thedeployment approach described above, it's likely not the best approach. First, the spack_cpu
and spack_gpu
deployment role accounts on Expanse cannot write into each others SPACK_ROOT trees from their respective deployment nodes due to how the Ceph keys are assigned to each individually. Moreover, even if this cross-write ability was possible, it's also probably not a great idea to have two Spack user accounts write into different SPACK_ROOT tree due to longer-term ownership concerns.
Instead, the more straightfoward approach is to deploy nvhpc@21.9
and the required NetCDF libraries and dependencies directly to the expanse/0.17.3/cpu/b
instance.
Pull request created and merged. https://github.com/sdsc/spack/pull/126
Specs deployed into production successfully.
[mkandes@login01 ~]$ module load nvhpc/21.9
[mkandes@login01 ~]$ module avail
------------- /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/nvhpc/21.9 -------------
hdf5/1.10.7/mwaa2bz netcdf-c/4.8.1/j4nyqvv netcdf-fortran/4.5.3/fg4qvqw
---------------- /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/Core ----------------
anaconda3/2021.05/q4munrg git-lfs/2.11.0/kmruniy pigz/2.6/bgymyil
aocc/3.2.0/io3s466 git/2.31.1/ldetm5y rclone/1.56.2/mldjorr
aria2/1.35.0/q32jtg2 intel/19.1.3.304/6pv46so sratoolkit/2.10.9/rn4humf
cmake/3.21.4/n5jtjsf matlab/2022b/lefe4oq subversion/1.14.0/qpzq6zs
entrezdirect/10.7.20190114/6pkkpx2 mercurial/5.8/qmgrjvl ucx/1.10.1/wla3unl
gcc/10.2.0/npcyll4 nvhpc/21.9/xxpthf5 (L)
gh/2.0.0/mkz3uxl parallel/20210922/sqru6rr
------------------------------------------- /cm/local/modulefiles --------------------------------------------
shared (L) singularitypro/3.11 (D) singularitypro/4.1.2 slurm/expanse/23.02.7 (L)
------------------------------------- /cm/shared/apps/access/modulefiles -------------------------------------
accessusage/0.5-1 cue-login-env
------------------------------------------- /usr/share/modulefiles -------------------------------------------
DefaultModules (L) cpu/0.17.3b (c,L,D) gpu/0.17.3b (g,D) nostack/0.17.3b (e,D)
cpu/0.15.4 (c) gpu/0.15.4 (g) nostack/0.15.4 (e)
------------------------------------------- /cm/shared/modulefiles -------------------------------------------
AMDuProf/3.4.475 sdsc/1.0 (L) slurm/expanse/23.02.7 (D)
default-environment slurm/expanse/current
Where:
L: Module is loaded
c: built natively for AMD Rome
e: not architecture specific
g: built natively for Intel Skylake
D: Default Module
Module defaults are chosen based on Find First Rules due to Name/Version/Version modules found in the module tree.
See https://lmod.readthedocs.io/en/latest/060_locating.html for details.
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
[mkandes@login01 ~]$
https://github.com/sdsc/spack/commit/484490cfe5cea6c0f5e386bbf036a4317c067a1a
Additional notes here: https://github.com/mkandes/notes/blob/main/2024/08/16.md
User confirmed deployed solution is working for them. Closing issue.
A user requires NetCDF libaries on Expanse that are compatible with the available PGI compilers for some custom code they intend to use a part of an upcoming class they begin teaching next week. Unfortuantely, the PGI compilers currently deployed in production on Expanse are only a part of the
expanse/0.15.4/gpu
andexpanse/0.17.3/gpu/b
Spack instances, whereas the user intends to only run this (non-GPU-acclerated) example code on Expanse's standard compute nodes. And in any case, there are currently no NetCDF libraries for the PGI compiler family in eitherexpanse/0.15.4/gpu
and/orexpanse/0.17.3/gpu/b
.I reviewed the user's code and attempted to compile it with the NetCDF libraries and other compiler combinations already available in the
expanse/0.15.4/cpu
andexpanse/0.17.3/cpu/b
production Spack instances. Unfortunately, the code utilzies some unusual, non-standard Fortran syntax only 'allowed' by the PGI compilers at this time. Both GCC and Intel compilers reported syntax-related issues, which while could be resolved by editing the user's code, this could also become somewhat time consuming depending how far this problem prevades the code --- an initial attempt was made to fix, but it became apparent it could be more significant of a change needed.A quick and dirty temporary fix for the user has been tested. We can deploy the NetCDF libraries into the
expanse/0.17.3/gpu/b
instance using thenvhpc/21.9
compiler, BUT build each package and their dependencies on the standard AMD compute nodes.https://access-ci.atlassian.net/browse/ATS-9795