Closed nwolter closed 1 year ago
Once the LOCAL_SPACK_ROOT
is configured, you can then temporarily setup the shared instance with the following modifications in your '~/.bashrc` file.
[mkandes@login02 ~]$ cat .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/mkandes/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/home/mkandes/miniconda3/etc/profile.d/conda.sh" ]; then
. "/home/mkandes/miniconda3/etc/profile.d/conda.sh"
else
export PATH="/home/mkandes/miniconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
declare -xr SHARED_SPACK_VERSION='0.17.3'
declare -xr SHARED_SPACK_INSTANCE_NAME='cpu'
declare -xr SHARED_SPACK_INSTANCE_VERSION='b'
declare -xr SHARED_SPACK_ROOT="/cm/shared/apps/spack/${SHARED_SPACK_VERSION}/${SHARED_SPACK_INSTANCE_NAME}/${SHARED_SPACK_INSTANCE_VERSION}"
declare -xr LOCAL_SPACK_NAMESPACE="${USER}"
declare -xr LOCAL_SPACK_TMPDIR='/tmp'
declare -xr LOCAL_SPACK_ROOT="${HOME}/.spack/${SHARED_SPACK_VERSION}/${SHARED_SPACK_INSTANCE_NAME}/${SHARED_SPACK_INSTANCE_VERSION}/${SHARED_SPACK_USER}"
. "${SHARED_SPACK_ROOT}/share/spack/setup-env.sh"
module use "${LOCAL_SPACK_ROOT}/share/spack/lmod/linux-rocky8-x86_64"
alias spack="spack --config-scope ${LOCAL_SPACK_ROOT}"
[mkandes@login02 ~]$
Upon starting a new shell session, you should have access to production Spack instance on Expanse, but it is now scoped to perform any installs into your LOCAL_SPACK_ROOT
in ~/.spack
.
[mkandes@login01 ~]$ spack --version
0.17.3-127-e6d631d16f
[mkandes@login01 ~]$ which spack
alias spack='spack --config-scope /home/mkandes/.spack/0.17.3/cpu/b/'
spack ()
{
: this is a shell function from: /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/setup-env.sh;
: the real spack script is here: /cm/shared/apps/spack/0.17.3/cpu/b/bin/spack;
_spack_shell_wrapper "$@";
return $?
}
[mkandes@login01 ~]$
Before installing a spark via Spack, let's check if the version we want is available.
[mkandes@login01 ~]$ spack info spark
Package: spark
Description:
Apache Spark is a fast and general engine for large-scale data
processing.
Homepage: https://spark.apache.org
Externally Detectable:
False
Tags:
None
Preferred version:
3.1.1 https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-without-hadoop.tgz
Safe versions:
3.1.1 https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-without-hadoop.tgz
3.0.2 https://archive.apache.org/dist/spark/spark-3.0.2/spark-3.0.2-bin-without-hadoop.tgz
3.0.1 https://archive.apache.org/dist/spark/spark-3.0.1/spark-3.0.1-bin-without-hadoop.tgz
2.4.7 https://archive.apache.org/dist/spark/spark-2.4.7/spark-2.4.7-bin-without-hadoop.tgz
2.3.0 https://archive.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-without-hadoop.tgz
2.1.0 https://archive.apache.org/dist/spark/spark-2.1.0/spark-2.1.0-bin-without-hadoop.tgz
2.0.2 https://archive.apache.org/dist/spark/spark-2.0.2/spark-2.0.2-bin-without-hadoop.tgz
2.0.0 https://archive.apache.org/dist/spark/spark-2.0.0/spark-2.0.0-bin-without-hadoop.tgz
1.6.2 https://archive.apache.org/dist/spark/spark-1.6.2/spark-1.6.2-bin-without-hadoop.tgz
1.6.1 https://archive.apache.org/dist/spark/spark-1.6.1/spark-1.6.1-bin-without-hadoop.tgz
1.6.0 https://archive.apache.org/dist/spark/spark-1.6.0/spark-1.6.0-bin-without-hadoop.tgz
Deprecated versions:
None
Variants:
Name [Default] When Allowed values Description
============== ==== ============== =================
hadoop [off] -- on, off Build with Hadoop
Installation Phases:
install
Build Dependencies:
hadoop java
Link Dependencies:
None
Run Dependencies:
hadoop java
Virtual Packages:
None
[mkandes@login01 ~]$
The latest version at the time of this writing is: https://archive.apache.org/dist/spark/spark-3.4.0. As such, let's also plan to upgrade our spark Spack package to include the latest downloadable tarball.
In this case, we'll likely want to install spark in the gcc@10.2.0
package dependency chain of expanse/0.17.3/cpu/b
as this would allow us to more closely integrate it with either the python or R package environments we support there via Spack. Let's check what namespace and/or package repo the latest version of spark is currently coming from ...
[mkandes@login01 ~]$ spack spec -lNt --reuse spark@3.1.1 % gcc@10.2.0 ~hadoop
Input spec
--------------------------------
[ ] .spark@3.1.1%gcc@10.2.0~hadoop
Concretized
--------------------------------
b6vwm6x [ ] builtin.spark@3.1.1%gcc@10.2.0 cflags="-O2 -march=native" cxxflags="-O2 -march=native" fflags="-O2 -march=native" ~hadoop arch=linux-rocky8-zen2
27cv2ps [b r ] ^builtin.openjdk@11.0.12_7%gcc@10.2.0 cflags="-O2 -march=native" cxxflags="-O2 -march=native" fflags="-O2 -march=native" arch=linux-rocky8-zen2
[mkandes@login01 ~]$
To upgrade the spark version, we'll need to modify the Preferred version
in the package.py
file to include the latest tarball and its hash. Note, however, before placing the new package in the sdsc spack package repo
[mkandes@login01 ~]$ ls $SHARED_SPACK_ROOT/var/spack/repos/sdsc/
packages repo.yaml
[mkandes@login01 ~]$
we can test it locally in my personal repo setup as part of the shared spack instance configuration.
[mkandes@login01 ~]$ ls $LOCAL_SPACK_ROOT/var/spack/repos/mkandes/
packages repo.yaml
[mkandes@login01 ~]$
Since the default package is currently coming from the builtin
repo, let's go ahead and make a copy to edit and use for our test build.
[mkandes@login01 ~]$ cd $LOCAL_SPACK_ROOT/var/spack/repos/mkandes/packages
[mkandes@login01 packages]$ pwd
/home/mkandes/.spack/0.17.3/cpu/b/var/spack/repos/mkandes/packages
[mkandes@login01 packages]$ ls $SHARED_SPACK_ROOT/var/spack/repos/builtin/packages/spark
package.py __pycache__
[mkandes@login01 packages]$ cp -rp $SHARED_SPACK_ROOT/var/spack/repos/builtin/packages/spark ./
[mkandes@login01 packages]$ pwd
/home/mkandes/.spack/0.17.3/cpu/b/var/spack/repos/mkandes/packages
[mkandes@login01 packages]$ ls
spark
[mkandes@login01 packages]$ cd spark/
[mkandes@login01 spark]$ rm __pycache__/
rm: cannot remove '__pycache__/': Is a directory
[mkandes@login01 spark]$ rm -rf __pycache__/
[mkandes@login01 spark]$ ls
package.py
[mkandes@login01 spark]$
Download and compute sha256sum
of the tarball.
mkandes@hardtack:~$ wget https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-without-hadoop.tgz
--2023-06-06 14:51:00-- https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-without-hadoop.tgz
Resolving archive.apache.org (archive.apache.org)... 65.108.204.189, 2a01:4f9:1a:a084::2
Connecting to archive.apache.org (archive.apache.org)|65.108.204.189|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 300271470 (286M) [application/x-gzip]
Saving to: ‘spark-3.4.0-bin-without-hadoop.tgz’
spark-3.4.0-bin-wit 100%[===================>] 286.36M 23.1MB/s in 20s
2023-06-06 14:51:21 (14.0 MB/s) - ‘spark-3.4.0-bin-without-hadoop.tgz’ saved [300271470/300271470]
mkandes@hardtack:~$ sha256sum spark-3.4.0-bin-without-hadoop.tgz
6c0366139dab03f9bfb788b27918e7e3d411adcfa5631de75ac91afac87fcf5f spark-3.4.0-bin-without-hadoop.tgz
mkandes@hardtack:~$
And then add the new version to the package.py
file.
version('3.4.0', sha256='6c0366139dab03f9bfb788b27918e7e3d411adcfa5631de75ac91afac87fcf5f')
And now let's see if the new spark package from my personal package repo now has precedence ...
[mkandes@login01 ~]$ spack info spark
Package: spark
Description:
Apache Spark is a fast and general engine for large-scale data
processing.
Homepage: https://spark.apache.org
Externally Detectable:
False
Tags:
None
Preferred version:
3.4.0 https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-without-hadoop.tgz
Safe versions:
3.4.0 https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-without-hadoop.tgz
3.1.1 https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-without-hadoop.tgz
3.0.2 https://archive.apache.org/dist/spark/spark-3.0.2/spark-3.0.2-bin-without-hadoop.tgz
3.0.1 https://archive.apache.org/dist/spark/spark-3.0.1/spark-3.0.1-bin-without-hadoop.tgz
2.4.7 https://archive.apache.org/dist/spark/spark-2.4.7/spark-2.4.7-bin-without-hadoop.tgz
2.3.0 https://archive.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-without-hadoop.tgz
2.1.0 https://archive.apache.org/dist/spark/spark-2.1.0/spark-2.1.0-bin-without-hadoop.tgz
2.0.2 https://archive.apache.org/dist/spark/spark-2.0.2/spark-2.0.2-bin-without-hadoop.tgz
2.0.0 https://archive.apache.org/dist/spark/spark-2.0.0/spark-2.0.0-bin-without-hadoop.tgz
1.6.2 https://archive.apache.org/dist/spark/spark-1.6.2/spark-1.6.2-bin-without-hadoop.tgz
1.6.1 https://archive.apache.org/dist/spark/spark-1.6.1/spark-1.6.1-bin-without-hadoop.tgz
1.6.0 https://archive.apache.org/dist/spark/spark-1.6.0/spark-1.6.0-bin-without-hadoop.tgz
Deprecated versions:
None
Variants:
Name [Default] When Allowed values Description
============== ==== ============== =================
hadoop [off] -- on, off Build with Hadoop
Installation Phases:
install
Build Dependencies:
hadoop java
Link Dependencies:
None
Run Dependencies:
hadoop java
Virtual Packages:
None
[mkandes@login01 ~]$
[mkandes@login01 ~]$ spack spec -lNt --reuse spark@3.4.0 % gcc@10.2.0 ~hadoop
Input spec
--------------------------------
[ ] .spark@3.4.0%gcc@10.2.0~hadoop
Concretized
--------------------------------
pwgwwdj [ ] mkandes.spark@3.4.0%gcc@10.2.0 cflags="-O2 -march=native" cxxflags="-O2 -march=native" fflags="-O2 -march=native" ~hadoop arch=linux-rocky8-zen2
27cv2ps [b r ] ^builtin.openjdk@11.0.12_7%gcc@10.2.0 cflags="-O2 -march=native" cxxflags="-O2 -march=native" fflags="-O2 -march=native" arch=linux-rocky8-zen2
[mkandes@login01 ~]$
Everything looks good. Now we need to construct a spec *.sh
build script to deploy spark in my HOME directory and into the shared instance config. Once that build test is complete, we can submit the new Spack package for spark and the spec build script to deploy it into production as a pull request to the sdsc/spack repo.
[mkandes@login01 ~]$ cp $SHARED_SPACK_ROOT/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/cmake@3.21.4.sh ./
[mkandes@login01 ~]$ mv cmake@3.21.4.sh spark@3.4.0.sh
[mkandes@login01 ~]$ vi spark@3.4.0.sh
[mkandes@login01 ~]$ sbatch spark@3.4.0.sh
Submitted batch job 22863818
[mkandes@login01 ~]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22863818 ind-share spark@3. mkandes R 0:03 1 exp-15-44
[mkandes@login01 ~]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22863818 ind-share spark@3. mkandes R 0:28 1 exp-15-44
[mkandes@login01 ~]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22863818 ind-share spark@3. mkandes R 0:51 1 exp-15-44
[mkandes@login01 ~]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22863818 ind-share spark@3. mkandes R 3:04 1 exp-15-44
[mkandes@login01 ~]$
The following is the spark@3.4.0.sh spec build script submitted to Slurm on Expanse above to install Spark within my local Spack instance in my HOME directory prior to submitting script to the sdsc/spack repo as a pull request.
#!/usr/bin/env bash
#SBATCH --job-name=spark@3.4.0
#SBATCH --account=use300
##SBATCH --reservation=rocky8u7_testing
#SBATCH --partition=ind-shared
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G
#SBATCH --time=00:30:00
#SBATCH --output=%x.o%j.%N
declare -xr LOCAL_TIME="$(date +'%Y%m%dT%H%M%S%z')"
declare -xir UNIX_TIME="$(date +'%s')"
declare -xr LOCAL_SCRATCH_DIR="/scratch/${USER}/job_${SLURM_JOB_ID}"
declare -xr TMPDIR="${LOCAL_SCRATCH_DIR}"
declare -xr SYSTEM_NAME='expanse'
declare -xr SPACK_VERSION='0.17.3'
declare -xr SPACK_INSTANCE_NAME='cpu'
declare -xr SPACK_INSTANCE_VERSION='b'
declare -xr SPACK_INSTANCE_DIR="/cm/shared/apps/spack/${SPACK_VERSION}/${SPACK_INSTANCE_NAME}/${SPACK_INSTANCE_VERSION}"
declare -xr SLURM_JOB_SCRIPT="$(scontrol show job ${SLURM_JOB_ID} | awk -F= '/Command=/{print $2}')"
declare -xr SLURM_JOB_MD5SUM="$(md5sum ${SLURM_JOB_SCRIPT})"
declare -xr SCHEDULER_MODULE='slurm'
echo "${UNIX_TIME} ${SLURM_JOB_ID} ${SLURM_JOB_MD5SUM} ${SLURM_JOB_DEPENDENCY}"
echo ""
cat "${SLURM_JOB_SCRIPT}"
module purge
module load "${SCHEDULER_MODULE}"
module list
. "${SPACK_INSTANCE_DIR}/share/spack/setup-env.sh"
shopt -s expand_aliases
source ~/.bashrc
declare -xr SPACK_PACKAGE='spark@3.4.0'
declare -xr SPACK_COMPILER='gcc@10.2.0'
declare -xr SPACK_VARIANTS='~hadoop'
declare -xr SPACK_DEPENDENCIES="^openjdk@1.8.0_265-b01/$(spack find --format '{hash:7}' openjdk@1.8.0_265-b01 % ${SPACK_COMPILER})"
declare -xr SPACK_SPEC="${SPACK_PACKAGE} % ${SPACK_COMPILER} ${SPACK_VARIANTS} ${SPACK_DEPENDENCIES}"
printenv
spack config get compilers
spack config get config
spack config get mirrors
spack config get modules
spack config get packages
spack config get repos
spack config get upstreams
time -p spack spec --long --namespaces --types --reuse "${SPACK_SPEC}"
if [[ "${?}" -ne 0 ]]; then
echo 'ERROR: spack concretization failed.'
exit 1
fi
time -p spack install --jobs "${SLURM_CPUS_PER_TASK}" --fail-fast --yes-to-all --reuse "${SPACK_SPEC}"
if [[ "${?}" -ne 0 ]]; then
echo 'ERROR: spack install failed.'
exit 1
fi
spack module lmod refresh -y
#sbatch --dependency="afterok:${SLURM_JOB_ID}" ''
sleep 30
Please note, the only custom changes needed to use the shared spack instance configuration setup in my ~/.bashrc
file are these two lines ...
shopt -s expand_aliases
source ~/.bashrc
which can be easily removed prior to production deployment --- and even if forgotten prior to deployment, should have no effect.
[mkandes@login01 ~]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
[mkandes@login01 ~]$ spack find -lvd spark@3.4.0
==> 1 installed package
-- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
pwgwwdj spark@3.4.0~hadoop
27cv2ps openjdk@11.0.12_7
[mkandes@login01 ~]$
[mkandes@login01 ~]$ module avail
---- /home/mkandes/.spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64 ----
gcc/10.2.0/py-numpy/1.21.3/53ovhtc
- /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/Core -
anaconda3/2021.05/q4munrg matlab/2022b/lefe4oq
aocc/3.2.0/io3s466 mercurial/5.8/qmgrjvl
aria2/1.35.0/q32jtg2 parallel/20210922/sqru6rr
entrezdirect/10.7.20190114/6pkkpx2 pigz/2.6/bgymyil
gcc/10.2.0/npcyll4 rclone/1.56.2/mldjorr
gh/2.0.0/mkz3uxl sratoolkit/2.10.9/rn4humf
git-lfs/2.11.0/kmruniy subversion/1.14.0/qpzq6zs
git/2.31.1/ldetm5y ucx/1.10.1/wla3unl
intel/19.1.3.304/6pv46so
Getting the module-based access working ...
[mkandes@login02 ~]$ spack module lmod refresh -y
==> Regenerating lmod module files
[mkandes@login02 ~]$ rm -rf .lmod.d/
[mkandes@login02 ~]$ module avail
---- /home/mkandes/.spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64 ----
gcc/10.2.0/py-numpy/1.21.3/53ovhtc gcc/10.2.0/spark/3.4.0/pwgwwdj
- /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/Core -
anaconda3/2021.05/q4munrg matlab/2022b/lefe4oq
aocc/3.2.0/io3s466 mercurial/5.8/qmgrjvl
aria2/1.35.0/q32jtg2 parallel/20210922/sqru6rr
entrezdirect/10.7.20190114/6pkkpx2 pigz/2.6/bgymyil
gcc/10.2.0/npcyll4 rclone/1.56.2/mldjorr
gh/2.0.0/mkz3uxl sratoolkit/2.10.9/rn4humf
git-lfs/2.11.0/kmruniy subversion/1.14.0/qpzq6zs
git/2.31.1/ldetm5y ucx/1.10.1/wla3unl
intel/19.1.3.304/6pv46so
Spark is now available from my local Spack install_tree ..
[mkandes@login02 ~]$ module load gcc/10.2.0
[mkandes@login02 ~]$ module load gcc/10.2.0/spark/3.4.0
[mkandes@login02 ~]$ module list
Currently Loaded Modules:
1) shared 5) DefaultModules
2) cpu/0.17.3b (c) 6) gcc/10.2.0/npcyll4
3) slurm/expanse/21.08.8 7) gcc/10.2.0/spark/3.4.0/pwgwwdj
4) sdsc/1.0
Where:
c: built natively for AMD Rome
[mkandes@login02 ~]$ which spark-submit
~/.spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/spark-3.4.0-pwgwwdjitbjz4gpj5nrckrc5sa75r2mn/bin/spark-submit
[mkandes@login02 ~]$
Redo with openjdk@1.8.0
... and moving to a local specs dir ...
[mkandes@login02 specs]$ sbatch spark@3.4.0.sh
Submitted batch job 22900951
[mkandes@login02 specs]$ squeeu -u $USER
-bash: squeeu: command not found
[mkandes@login02 specs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22900951 ind-share spark@3. mkandes R 0:55 1 exp-15-05
[mkandes@login02 specs]$ vi spark@3.4.0.sh
[mkandes@login02 specs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22900951 ind-share spark@3. mkandes R 2:42 1 exp-15-05
[mkandes@login02 specs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
[mkandes@login02 specs]$ spack find -lvd spark@3.4.0
==> 1 installed package
-- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
rksnhif spark@3.4.0~hadoop
cgiyxcb openjdk@1.8.0_265-b01
[mkandes@login02 specs]$ module avail
---- /home/mkandes/.spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64 ----
gcc/10.2.0/py-numpy/1.21.3/53ovhtc gcc/10.2.0/spark/3.4.0/rksnhif
- /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/Core -
anaconda3/2021.05/q4munrg matlab/2022b/lefe4oq
aocc/3.2.0/io3s466 mercurial/5.8/qmgrjvl
aria2/1.35.0/q32jtg2 parallel/20210922/sqru6rr
entrezdirect/10.7.20190114/6pkkpx2 pigz/2.6/bgymyil
gcc/10.2.0/npcyll4 rclone/1.56.2/mldjorr
gh/2.0.0/mkz3uxl sratoolkit/2.10.9/rn4humf
git-lfs/2.11.0/kmruniy subversion/1.14.0/qpzq6zs
git/2.31.1/ldetm5y ucx/1.10.1/wla3unl
intel/19.1.3.304/6pv46so
---------------------------- /cm/local/modulefiles -----------------------------
cmjob openmpi/mlnx/gcc/64/4.1.5a1 singularitypro/3.9
lua/5.3.5 shared (L) slurm/expanse/21.08.8 (L)
---------------------- /cm/shared/apps/access/modulefiles ----------------------
accessusage/0.5-1 cue-login-env
---------------------------- /usr/share/modulefiles ----------------------------
[mkandes@login02 specs]$ ls
spark@3.4.0.o22900951.exp-15-05 spark@3.4.0.sh
[mkandes@login02 specs]$ pwd
/home/mkandes/.spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs
[mkandes@login02 specs]$
Let's do this one more time ... forgot +hadoop
is required, unless you configure spark to use an external version already running on your system.
[mkandes@login02 specs]$ spack find -lvd spark@3.4.0
==> 1 installed package
-- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
rksnhif spark@3.4.0~hadoop
cgiyxcb openjdk@1.8.0_265-b01
[mkandes@login02 specs]$ spack uninstall spark@3.4.0
==> The following packages will be uninstalled:
-- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
rksnhif spark@3.4.0
==> Do you want to proceed? [y/N] y
==> Successfully uninstalled spark@3.4.0%gcc@10.2.0~hadoop arch=linux-rocky8-zen2/rksnhif
[mkandes@login02 specs]$
[mkandes@login02 specs]$ cat spark@3.4.0.sh
#!/usr/bin/env bash
#SBATCH --job-name=spark@3.4.0
#SBATCH --account=use300
##SBATCH --reservation=rocky8u7_testing
#SBATCH --partition=ind-shared
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G
#SBATCH --time=00:30:00
#SBATCH --output=%x.o%j.%N
declare -xr LOCAL_TIME="$(date +'%Y%m%dT%H%M%S%z')"
declare -xir UNIX_TIME="$(date +'%s')"
declare -xr LOCAL_SCRATCH_DIR="/scratch/${USER}/job_${SLURM_JOB_ID}"
declare -xr TMPDIR="${LOCAL_SCRATCH_DIR}"
declare -xr SYSTEM_NAME='expanse'
declare -xr SPACK_VERSION='0.17.3'
declare -xr SPACK_INSTANCE_NAME='cpu'
declare -xr SPACK_INSTANCE_VERSION='b'
declare -xr SPACK_INSTANCE_DIR="/cm/shared/apps/spack/${SPACK_VERSION}/${SPACK_INSTANCE_NAME}/${SPACK_INSTANCE_VERSION}"
declare -xr SLURM_JOB_SCRIPT="$(scontrol show job ${SLURM_JOB_ID} | awk -F= '/Command=/{print $2}')"
declare -xr SLURM_JOB_MD5SUM="$(md5sum ${SLURM_JOB_SCRIPT})"
declare -xr SCHEDULER_MODULE='slurm'
echo "${UNIX_TIME} ${SLURM_JOB_ID} ${SLURM_JOB_MD5SUM} ${SLURM_JOB_DEPENDENCY}"
echo ""
cat "${SLURM_JOB_SCRIPT}"
module purge
module load "${SCHEDULER_MODULE}"
module list
. "${SPACK_INSTANCE_DIR}/share/spack/setup-env.sh"
shopt -s expand_aliases
source ~/.bashrc
declare -xr SPACK_PACKAGE='spark@3.4.0'
declare -xr SPACK_COMPILER='gcc@10.2.0'
declare -xr SPACK_VARIANTS='+hadoop'
declare -xr SPACK_DEPENDENCIES="^openjdk@11.0.12_7/$(spack find --format '{hash:7}' openjdk@11.0.12_7 % ${SPACK_COMPILER})"
declare -xr SPACK_SPEC="${SPACK_PACKAGE} % ${SPACK_COMPILER} ${SPACK_VARIANTS} ${SPACK_DEPENDENCIES}"
printenv
spack config get compilers
spack config get config
spack config get mirrors
spack config get modules
spack config get packages
spack config get repos
spack config get upstreams
time -p spack spec --long --namespaces --types --reuse "${SPACK_SPEC}"
if [[ "${?}" -ne 0 ]]; then
echo 'ERROR: spack concretization failed.'
exit 1
fi
time -p spack install --jobs "${SLURM_CPUS_PER_TASK}" --fail-fast --yes-to-all --reuse "${SPACK_SPEC}"
if [[ "${?}" -ne 0 ]]; then
echo 'ERROR: spack install failed.'
exit 1
fi
spack module lmod refresh -y
#sbatch --dependency="afterok:${SLURM_JOB_ID}" ''
sleep 30
[mkandes@login02 specs]$ vi spark@3.4.0.sh
[mkandes@login02 specs]$ rm spark@3.4.0.o22901977.exp-15-45
[mkandes@login02 specs]$ sbatch spark@3.4.0.sh
Submitted batch job 22902088
[mkandes@login02 specs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22902088 ind-share spark@3. mkandes R 7:35 1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22902088 ind-share spark@3. mkandes R 17:35 1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22902088 ind-share spark@3. mkandes R 19:11 1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22902088 ind-share spark@3. mkandes R 30:34 1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22902088 ind-share spark@3. mkandes R 40:21 1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
22902088 ind-share spark@3. mkandes R 41:38 1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
[mkandes@login02 specs]$ ls -lahtr
total 28K
drwxr-xr-x 3 mkandes use300 3 Jun 9 07:05 ..
-rw-r--r-- 1 mkandes use300 2.2K Jun 9 08:10 spark@3.4.0.sh
drwxr-xr-x 2 mkandes use300 4 Jun 9 08:10 .
-rw-r--r-- 1 mkandes use300 26K Jun 9 08:57 spark@3.4.0.o22902088.exp-15-02
[mkandes@login02 specs]$
Picking back up where I left off last month ...
[mkandes@login01 ~]$ ls -lahtr /home/mkandes/.spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs
total 28K
drwxr-xr-x 3 mkandes use300 3 Jun 9 07:05 ..
-rw-r--r-- 1 mkandes use300 2.2K Jun 9 08:10 spark@3.4.0.sh
drwxr-xr-x 2 mkandes use300 4 Jun 9 08:10 .
-rw-r--r-- 1 mkandes use300 26K Jun 9 08:57 spark@3.4.0.o22902088.exp-15-02
[mkandes@login01 ~]$ module avail
---- /home/mkandes/.spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64 ----
gcc/10.2.0/py-numpy/1.21.3/53ovhtc gcc/10.2.0/spark/3.4.0/po6mvtn
- /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/Core -
anaconda3/2021.05/q4munrg matlab/2022b/lefe4oq
aocc/3.2.0/io3s466 mercurial/5.8/qmgrjvl
aria2/1.35.0/q32jtg2 parallel/20210922/sqru6rr
entrezdirect/10.7.20190114/6pkkpx2 pigz/2.6/bgymyil
gcc/10.2.0/npcyll4 rclone/1.56.2/mldjorr
gh/2.0.0/mkz3uxl sratoolkit/2.10.9/rn4humf
git-lfs/2.11.0/kmruniy subversion/1.14.0/qpzq6zs
git/2.31.1/ldetm5y ucx/1.10.1/wla3unl
intel/19.1.3.304/6pv46so
---------------------------- /cm/local/modulefiles -----------------------------
cmjob openmpi/mlnx/gcc/64/4.1.5a1 singularitypro/3.9
lua/5.3.5 shared (L) slurm/expanse/21.08.8 (L)
---------------------- /cm/shared/apps/access/modulefiles ----------------------
accessusage/0.5-1 cue-login-env
---------------------------- /usr/share/modulefiles ----------------------------
[mkandes@login01 ~]$
... the next step would be to run tests against the locally deployed application before submitting it upstream as a pull request. Unfortunately, here in the case of Spark, the application testing is/was more complicated and is not working as intended and/or expected at this time. However, I will continue on and complete the deployment as an initial example for documenting the process, which will then be written up in the sdsc/spack project's documentation.
Before updating your personal fork of the sdsc/spack repository with the spec build script, you'll need to first re-sync it with the sdsc/spack as its upstream. Using a command-line approach on Expanse, this workflow is as follows ...
[mkandes@login02 repos]$ git clone https://github.com/mkandes/spack.git
Cloning into 'spack'...
remote: Enumerating objects: 463470, done.
remote: Total 463470 (delta 0), reused 0 (delta 0), pack-reused 463470
Receiving objects: 100% (463470/463470), 370.43 MiB | 37.79 MiB/s, done.
Resolving deltas: 100% (203948/203948), done.
Updating files: 100% (10124/10124), done.
[mkandes@login02 repos]$ ls
spack
[mkandes@login02 repos]$ mv spack/ mkandes/
[mkandes@login02 repos]$ ls
mkandes
[mkandes@login02 repos]$ pwd
/home/mkandes/software/spack/repos
[mkandes@login02 repos]$ cd mkandes/
[mkandes@login02 mkandes]$ ls
bin DEPLOYMENT.md LICENSE-MIT README.md
CHANGELOG.md etc NOTICE SECURITY.md
CONTRIBUTING.md lib pyproject.toml share
COPYRIGHT LICENSE-APACHE pytest.ini var
[mkandes@login02 mkandes]$ git log
commit e6d631d16ff491cdbac076f05b0da1ded43590ff (HEAD -> sdsc-0.17.3, origin/sdsc-0.17.3, origin/HEAD)
Merge: 439fba6469 298234311b
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jun 5 11:37:43 2023 -0700
Merge pull request #82 from mkandes/sdsc-0.17.3-yaml-modules-sdsc-gh-80
Modify modules.yaml for expanse/0.17.3/cpu/b to whitelist py-jupyter-…
commit 298234311b4e2b75bec80b9335f6e538941100e4 (origin/sdsc-0.17.3-yaml-modules-sdsc-gh-80)
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jun 5 11:21:52 2023 -0700
Modify modules.yaml for expanse/0.17.3/cpu/b to whitelist py-jupyter-core
This minor change allows us to explicitly generate an lmod modulefile
for the py-jupyter-core Spack package that was installed implicitly as
a dependency of py-jupyterlab.
See https://github.com/sdsc/spack/issues/80
[mkandes@login02 mkandes]$ git fetch origin
[mkandes@login02 mkandes]$ git branch
* sdsc-0.17.3
[mkandes@login02 mkandes]$ git remote add upstream https://github.com/sdsc/spack.git
[mkandes@login02 mkandes]$ git fetch --tags upstream
remote: Enumerating objects: 1, done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 1
Unpacking objects: 100% (1/1), 687 bytes | 3.00 KiB/s, done.
From https://github.com/sdsc/spack
* [new branch] FASTMath/ATPESC18 -> upstream/FASTMath/ATPESC18
* [new branch] FASTMath/ATPESC19 -> upstream/FASTMath/ATPESC19
* [new branch] PR/openblas -> upstream/PR/openblas
* [new branch] PR/pymock -> upstream/PR/pymock
* [new branch] add-miniconda3@4.8.2-and-revert-4.7.12.1 -> upstream/add-miniconda3@4.8.2-and-revert-4.7.12.1
* [new branch] amklinv/amanzi -> upstream/amklinv/amanzi
* [new branch] ascent -> upstream/ascent
* [new branch] autotools-cflags -> upstream/autotools-cflags
* [new branch] barry/ceed -> upstream/barry/ceed
* [new branch] boost-intel-toolset -> upstream/boost-intel-toolset
* [new branch] bugfix/buildable-true-override-virtual -> upstream/bugfix/buildable-true-override-virtual
* [new branch] bugfix/bz2-libs -> upstream/bugfix/bz2-libs
* [new branch] bugfix/compare-abstract-specs -> upstream/bugfix/compare-abstract-specs
* [new branch] bugfix/cray-no-module-preloaded -> upstream/bugfix/cray-no-module-preloaded
* [new branch] bugfix/env-activate-only-installed-concrete-specs -> upstream/bugfix/env-activate-only-installed-concrete-specs
* [new branch] bugfix/improved-uid-detection -> upstream/bugfix/improved-uid-detection
* [new branch] bugfix/m4_sierra_gcc_2 -> upstream/bugfix/m4_sierra_gcc_2
* [new branch] bugfix/multiarch-detection -> upstream/bugfix/multiarch-detection
* [new branch] bugfix/ordered-compiler-names -> upstream/bugfix/ordered-compiler-names
* [new branch] bugfix/sbang-filter-non-ascii-python-3 -> upstream/bugfix/sbang-filter-non-ascii-python-3
* [new branch] bugfix/setup-env -> upstream/bugfix/setup-env
* [new branch] bugfix/shasta-hotfix-live-branch -> upstream/bugfix/shasta-hotfix-live-branch
* [new branch] bugfix/target-args-cray-compilers -> upstream/bugfix/target-args-cray-compilers
* [new branch] cdash/xsdk -> upstream/cdash/xsdk
* [new branch] chain-doc-minor-improvement -> upstream/chain-doc-minor-improvement
* [new branch] concretizer -> upstream/concretizer
* [new branch] cray-libsci-and-ver -> upstream/cray-libsci-and-ver
* [new branch] dependabot/github_actions/actions/setup-python-4.6.0 -> upstream/dependabot/github_actions/actions/setup-python-4.6.0
* [new branch] dependabot/github_actions/docker/build-push-action-4.0.0 -> upstream/dependabot/github_actions/docker/build-push-action-4.0.0
* [new branch] dependabot/github_actions/docker/login-action-2.1.0 -> upstream/dependabot/github_actions/docker/login-action-2.1.0
* [new branch] dependabot/github_actions/docker/setup-buildx-action-2.5.0 -> upstream/dependabot/github_actions/docker/setup-buildx-action-2.5.0
* [new branch] dependabot/github_actions/docker/setup-qemu-action-2.1.0 -> upstream/dependabot/github_actions/docker/setup-qemu-action-2.1.0
* [new branch] develop -> upstream/develop
* [new branch] docs/environment-scopes -> upstream/docs/environment-scopes
* [new branch] documentation/asp_based_solver -> upstream/documentation/asp_based_solver
* [new branch] eaf/200402-SpackEnvironmentDocs -> upstream/eaf/200402-SpackEnvironmentDocs
* [new branch] efischer/190121-SetupEnv -> upstream/efischer/190121-SetupEnv
* [new branch] efischer/190923-MultiPackagePrototype -> upstream/efischer/190923-MultiPackagePrototype
* [new branch] efischer/docs2 -> upstream/efischer/docs2
* [new branch] features/add-buildcache-list-to-database -> upstream/features/add-buildcache-list-to-database
* [new branch] features/anonymous_specs-fix -> upstream/features/anonymous_specs-fix
* [new branch] features/arch-swapping -> upstream/features/arch-swapping
* [new branch] features/cached-cmake-build-system -> upstream/features/cached-cmake-build-system
* [new branch] features/cflags-resuse -> upstream/features/cflags-resuse
* [new branch] features/cmake_deps -> upstream/features/cmake_deps
* [new branch] features/compil -> upstream/features/compil
* [new branch] features/detect-and-swap-current-python -> upstream/features/detect-and-swap-current-python
* [new branch] features/env-improvements-release -> upstream/features/env-improvements-release
* [new branch] features/env-pre-cmd -> upstream/features/env-pre-cmd
* [new branch] features/find-linux-compilers-by-module -> upstream/features/find-linux-compilers-by-module
* [new branch] features/fix-cc-wrapper -> upstream/features/fix-cc-wrapper
* [new branch] features/improved-python-detection -> upstream/features/improved-python-detection
* [new branch] features/libs-command -> upstream/features/libs-command
* [new branch] features/make-platform-command-line-option -> upstream/features/make-platform-command-line-option
* [new branch] features/more-coverage -> upstream/features/more-coverage
* [new branch] features/multi-download -> upstream/features/multi-download
* [new branch] features/parallel-mirror -> upstream/features/parallel-mirror
* [new branch] features/rstudio -> upstream/features/rstudio
* [new branch] features/sandbox-builds -> upstream/features/sandbox-builds
* [new branch] features/setup_py -> upstream/features/setup_py
* [new branch] features/shebang-warning-to-msg -> upstream/features/shebang-warning-to-msg
* [new branch] features/solver -> upstream/features/solver
* [new branch] features/spack-mirror-verify -> upstream/features/spack-mirror-verify
* [new branch] features/spack-test-add-show-subcommand -> upstream/features/spack-test-add-show-subcommand
* [new branch] features/spack-test-hdf5-fix -> upstream/features/spack-test-hdf5-fix
* [new branch] features/spack-test-openmpi -> upstream/features/spack-test-openmpi
* [new branch] features/spack-test-openmpi-add-tests -> upstream/features/spack-test-openmpi-add-tests
* [new branch] features/stat_coral_test -> upstream/features/stat_coral_test
* [new branch] features/tty-output -> upstream/features/tty-output
* [new branch] features/view-with-env -> upstream/features/view-with-env
* [new branch] features/virtual-packages -> upstream/features/virtual-packages
* [new branch] fish-shell-1 -> upstream/fish-shell-1
* [new branch] fix_cc -> upstream/fix_cc
* [new branch] flake8-sha256 -> upstream/flake8-sha256
* [new branch] fugaku-cross-compilation -> upstream/fugaku-cross-compilation
* [new branch] gh-pages -> upstream/gh-pages
* [new branch] github/pr_templates -> upstream/github/pr_templates
* [new branch] implicit-variables-setup -> upstream/implicit-variables-setup
* [new branch] legion-update -> upstream/legion-update
* [new branch] libfabric-191-seems-to-work-on-darwin -> upstream/libfabric-191-seems-to-work-on-darwin
* [new branch] mfem -> upstream/mfem
* [new branch] mfem-updates -> upstream/mfem-updates
* [new branch] mfu-cmake -> upstream/mfu-cmake
* [new branch] mpbelhorn-aomp-3.9.0-wip-patch -> upstream/mpbelhorn-aomp-3.9.0-wip-patch
* [new branch] package/archer -> upstream/package/archer
* [new branch] packages/gsl-external-cblas -> upstream/packages/gsl-external-cblas
* [new branch] packages/pruners-mpi -> upstream/packages/pruners-mpi
* [new branch] packages/tulip -> upstream/packages/tulip
* [new branch] parse-oneapi-version -> upstream/parse-oneapi-version
* [new branch] pipelines-fix-develop-ambiguity -> upstream/pipelines-fix-develop-ambiguity
* [new branch] pipelines-improve-ci-job-label -> upstream/pipelines-improve-ci-job-label
* [new branch] proto/sys-packages-old -> upstream/proto/sys-packages-old
* [new branch] py-cffi-fix-checksum -> upstream/py-cffi-fix-checksum
* [new branch] py26-bootstrap -> upstream/py26-bootstrap
* [new branch] release_pipeline -> upstream/release_pipeline
* [new branch] releases/v0.10.0 -> upstream/releases/v0.10.0
* [new branch] releases/v0.11.0 -> upstream/releases/v0.11.0
* [new branch] releases/v0.11.1 -> upstream/releases/v0.11.1
* [new branch] releases/v0.11.2 -> upstream/releases/v0.11.2
* [new branch] releases/v0.12 -> upstream/releases/v0.12
* [new branch] releases/v0.13 -> upstream/releases/v0.13
* [new branch] releases/v0.13-hwlloc-master -> upstream/releases/v0.13-hwlloc-master
* [new branch] releases/v0.14 -> upstream/releases/v0.14
* [new branch] releases/v0.15 -> upstream/releases/v0.15
* [new branch] releases/v0.16 -> upstream/releases/v0.16
* [new branch] retry_serial -> upstream/retry_serial
* [new branch] revert-4.7.12.1 -> upstream/revert-4.7.12.1
* [new branch] sdsc-0.15.4 -> upstream/sdsc-0.15.4
* [new branch] sdsc-0.17.1 -> upstream/sdsc-0.17.1
* [new branch] sdsc-0.17.2 -> upstream/sdsc-0.17.2
* [new branch] sdsc-0.17.3 -> upstream/sdsc-0.17.3
* [new branch] sdsc-0.18.0 -> upstream/sdsc-0.18.0
* [new branch] sdsc-0.19.1 -> upstream/sdsc-0.19.1
* [new branch] separate-wrapper-decorator -> upstream/separate-wrapper-decorator
* [new branch] shorten-install-status-help -> upstream/shorten-install-status-help
* [new branch] silo-qt4 -> upstream/silo-qt4
* [new branch] spacktivate -> upstream/spacktivate
* [new branch] spec-language-ideas -> upstream/spec-language-ideas
* [new branch] spec-parse-inequalities -> upstream/spec-parse-inequalities
* [new branch] spec-target-flag-option -> upstream/spec-target-flag-option
* [new branch] stacks-debugging -> upstream/stacks-debugging
* [new branch] test-gitlab-check-stays-pending -> upstream/test-gitlab-check-stays-pending
* [new branch] test/v0.13-with-fast-environments -> upstream/test/v0.13-with-fast-environments
* [new branch] testing/wip_xsdk -> upstream/testing/wip_xsdk
* [new branch] tests/vtk-m-smoke-test-fix -> upstream/tests/vtk-m-smoke-test-fix
* [new branch] topic-libSplash160 -> upstream/topic-libSplash160
* [new branch] tutorials/advanced_packaging -> upstream/tutorials/advanced_packaging
* [new branch] update/ipopt -> upstream/update/ipopt
* [new branch] v-dobrev/extend-default-spec-queries -> upstream/v-dobrev/extend-default-spec-queries
* [new branch] vendor-clingo -> upstream/vendor-clingo
* [new branch] z3-bootstrap-2.6 -> upstream/z3-bootstrap-2.6
* [new branch] z3-concretizer -> upstream/z3-concretizer
[mkandes@login02 mkandes]$ git merge upstream/sdsc-0.17.3
Updating e6d631d16f..c33518253b
Fast-forward
etc/spack/sdsc/expanse/0.17.3/gpu/b/yamls/modules.yaml | 1 +
1 file changed, 1 insertion(+)
[mkandes@login02 mkandes]$ git log
commit c33518253b5cd6096e34c63c6fea4a7cd6618c7b (HEAD -> sdsc-0.17.3, upstream/sdsc-0.17.3)
Merge: e6d631d16f 51d103d923
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jun 5 14:08:54 2023 -0700
Merge pull request #83 from mkandes/sdsc-0.17.3-gh-81-yaml-modules-whitelist
Modify modules.yaml for expanse/0.17.3/gpu/b to whitelist py-jupyter-…
commit 51d103d923b635f0b7908b4c8a3fa69815e9a655 (origin/sdsc-0.17.3-gh-81-yaml-modules-whitelist)
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jun 5 14:07:21 2023 -0700
Modify modules.yaml for expanse/0.17.3/gpu/b to whitelist py-jupyter-core
See https://github.com/sdsc/spack/issues/81
commit e6d631d16ff491cdbac076f05b0da1ded43590ff (origin/sdsc-0.17.3, origin/HEAD)
Merge: 439fba6469 298234311b
Author: Marty Kandes <mkandes@sdsc.edu>
[mkandes@login02 mkandes]$ ls
bin DEPLOYMENT.md LICENSE-MIT README.md
CHANGELOG.md etc NOTICE SECURITY.md
CONTRIBUTING.md lib pyproject.toml share
COPYRIGHT LICENSE-APACHE pytest.ini var
[mkandes@login02 mkandes]$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com':
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 707 bytes | 707.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/mkandes/spack.git
e6d631d16f..c33518253b sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login02 mkandes]$ git push --tags
Everything up-to-date
[mkandes@login02 mkandes]$
We will also provide in the future a workflow using the GitHub web interface as well.
Next, copy both the new spec build script and test build output files to the correct location in your fork of the sdsc/spack repository ...
[mkandes@login02 specs]$ pwd
/home/mkandes/.spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs
[mkandes@login02 specs]$ ls
spark@3.4.0.o22902088.exp-15-02 spark@3.4.0.sh
[mkandes@login02 specs]$
[mkandes@login02 gcc@10.2.0]$ pwd
/home/mkandes/software/spack/repos/mkandes/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0
[mkandes@login02 gcc@10.2.0]$ cp -rp /home/mkandes/.spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/spark@3.4.0.sh ./
[mkandes@login02 gcc@10.2.0]$ cp -rp /home/mkandes/.spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/spark@3.4.0.o22902088.exp-15-02 ./
[mkandes@login02 gcc@10.2.0]$ ls | grep spark
spark@3.4.0.o22902088.exp-15-02
spark@3.4.0.sh
[mkandes@login02 gcc@10.2.0]$
Once the new spec build script and the test build output file are placed in the correct location, add the build script to the end of existing package dependency chain for the production deployment.
[mkandes@login02 gcc@10.2.0]$ vi py-htseq@0.11.2.sh
[mkandes@login02 gcc@10.2.0]$ grep spark py-htseq@0.11.2.sh
sbatch --dependency="afterok:${SLURM_JOB_ID}" 'spark@3.4.0.sh'
[mkandes@login02 gcc@10.2.0]$
See https://github.com/mkandes/spack/blob/sdsc-0.17.3/etc/spack/sdsc/expanse/0.17.3/cpu/b/SPECS.md
OR
... check the deployed instance itself for the last known set of deployed build scripts to find the appropriate location to link in the new build script into the dependency chain ...
[mkandes@login02 gcc@10.2.0]$ pwd
/cm/shared/apps/spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0
[mkandes@login02 gcc@10.2.0]$ ls -lahtr | tail -n 30
-rw-r--r-- 1 spack_cpu spack 32K Apr 16 00:32 jellyfish@2.2.7.o21704016.exp-15-56
-rw-r--r-- 1 spack_cpu spack 42K Apr 16 00:36 salmon@1.4.0.o21704017.exp-15-56
-rw-r--r-- 1 spack_cpu spack 36K Apr 16 00:39 iq-tree@2.1.3.o21704079.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.3K Apr 16 10:11 iq-tree@2.1.3.sh
-rw-r--r-- 1 spack_cpu spack 22K Apr 16 10:17 hisat2@2.2.0.o21711278.exp-15-56
-rw-r--r-- 1 spack_cpu spack 91K Apr 16 10:20 perl-bioperl@1.7.6.o21711315.exp-15-56
-rw-r--r-- 1 spack_cpu spack 57K Apr 16 10:21 py-pysam@0.15.3.o21711330.exp-15-56
-rw-r--r-- 1 spack_cpu spack 30K Apr 16 11:54 ucx@1.10.1.o21712584.exp-15-56
-rw-r--r-- 1 spack_cpu spack 34K Apr 16 12:07 openmpi@4.1.3.o21712606.exp-15-56
-rw-r--r-- 1 spack_cpu spack 3.1K Apr 17 07:02 openmpi@4.1.1.sh
-rw-r--r-- 1 spack_cpu spack 3.1K Apr 17 07:02 openmpi@4.1.3.sh
-rw-r--r-- 1 spack_cpu spack 32K Apr 17 07:16 openmpi@4.1.1.o21730146.exp-15-56
-rw-r--r-- 1 spack_cpu spack 36K Apr 17 07:16 openmpi@3.1.6.o21730154.exp-15-56
drwxr-sr-x 2 spack_cpu spack 115 Apr 17 07:30 openmpi@4.1.3
-rw-r--r-- 1 spack_cpu spack 2.5K Apr 17 07:47 mvapich2@2.3.7.sh
-rw-r--r-- 1 spack_cpu spack 2.0K Apr 17 07:48 intel-mpi@2019.10.317.sh
-rw-r--r-- 1 spack_cpu spack 35K Apr 17 07:58 openmpi@3.1.6.o21730681.exp-15-56
-rw-r--r-- 1 spack_cpu spack 29K Apr 17 08:00 mvapich2@2.3.7.o21730613.exp-15-56
-rw-r--r-- 1 spack_cpu spack 3.5K Apr 17 08:07 openmpi@3.1.6.sh
-rw-r--r-- 1 spack_cpu spack 35K Apr 17 08:17 openmpi@3.1.6.o21730823.exp-15-56
-rw-r--r-- 1 spack_cpu spack 22K Apr 17 18:44 intel-mpi@2019.10.317.o21739793.exp-15-56
drwxr-sr-x 2 spack_cpu spack 47 Apr 17 19:09 mvapich2@2.3.7
drwxr-sr-x 2 spack_cpu spack 45 Apr 17 19:43 intel-mpi@2019.10.317
drwxr-sr-x 7 spack_cpu spack 43 Apr 20 08:23 ..
drwxr-sr-x 2 spack_cpu spack 1 Apr 21 08:42 openmpi@4.1.1
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 08:53 openblas@0.3.18-omp.sh
-rw-r--r-- 1 spack_cpu spack 2.2K Apr 21 08:54 openblas@0.3.18.sh
-rw-r--r-- 1 spack_cpu spack 27K Apr 21 08:56 openblas@0.3.18-i64.o21819638.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 09:00 openblas@0.3.18-i64.sh
drwxr-sr-x 7 spack_cpu spack 264 Apr 21 09:00 .
[mkandes@login02 gcc@10.2.0]$
We are working on some additional tooling to make this discovery process simpler in the future. For now, best effort to keep things organized is what we aim to do. Using the SPECS.md documentation on the dependency chains for this example.
When ready, commit the changes to your fork. Please note, however, in this case this is a direct commit on the sdsc-0.17.3
deployment branch. In general, you should first created a spec branch off of the deployment branch before adding and committing these changes to the fork.
[mkandes@login02 gcc@10.2.0]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: py-htseq@0.11.2.sh
Untracked files:
(use "git add <file>..." to include in what will be committed)
spark@3.4.0.o22902088.exp-15-02
spark@3.4.0.sh
It took 3.10 seconds to enumerate untracked files. 'status -uno'
may speed it up, but you have to be careful not to forget to add
new files yourself (see 'git help status').
no changes added to commit (use "git add" and/or "git commit -a")
[mkandes@login02 gcc@10.2.0]$ git add -u
[mkandes@login02 gcc@10.2.0]$ git add spark@3.4.0.o22902088.exp-15-02
[mkandes@login02 gcc@10.2.0]$ git add spark@3.4.0.sh
[mkandes@login02 gcc@10.2.0]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: py-htseq@0.11.2.sh
new file: spark@3.4.0.o22902088.exp-15-02
new file: spark@3.4.0.sh
[mkandes@login02 gcc@10.2.0]$ git commit
[sdsc-0.17.3 90f124605b] Add spark@3.4.0 % gcc@10.2.0 to expanse/0.17.3/cpu/b
3 files changed, 726 insertions(+), 1 deletion(-)
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.o22902088.exp-15-02
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
[mkandes@login02 gcc@10.2.0]$ git log
commit 90f124605b39fb79339dd2efe492ec1437b20b79 (HEAD -> sdsc-0.17.3)
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jul 10 16:18:34 2023 -0700
Add spark@3.4.0 % gcc@10.2.0 to expanse/0.17.3/cpu/b
Push the changes back to your fork.
[mkandes@login02 gcc@10.2.0]$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com':
Enumerating objects: 25, done.
Counting objects: 100% (25/25), done.
Delta compression using up to 64 threads
Compressing objects: 100% (10/10), done.
Writing objects: 100% (14/14), 9.04 KiB | 4.52 MiB/s, done.
Total 14 (delta 6), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (6/6), completed with 5 local objects.
To https://github.com/mkandes/spack.git
c33518253b..90f124605b sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login02 gcc@10.2.0]$
Create pull request, review pull request (with other team members), merge pull request with sdsc/spack deployment branch prior to production deployment.
When ready to deploy package into production, login to role account, start interactive session for reserved node, pull down changes from sdsc/spack repo and deploy ...
[spack_cpu@login02 ~]$ srun --partition=ind-shared --reservation=root_73 --account=use300 --nodes=1 --nodelist=exp-15-56 --ntasks-per-node=1 --cpus-per-task=16 --mem=32G --time=12:00:00 --pty --wait=0 /bin/bash
[spack_cpu@exp-15-56 ~]$ cd /cm/shared/apps/spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/
[spack_cpu@exp-15-56 gcc@10.2.0]$ git stash
No local changes to save
[spack_cpu@exp-15-56 gcc@10.2.0]$ git pull
remote: Enumerating objects: 27, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 27 (delta 9), reused 15 (delta 8), pack-reused 9
Unpacking objects: 100% (27/27), 11.38 KiB | 315.00 KiB/s, done.
From https://github.com/sdsc/spack
e6d631d16f..e9bb2189ca sdsc-0.17.3 -> origin/sdsc-0.17.3
Updating e6d631d16f..e9bb2189ca
Fast-forward
.../cpu/b/specs/gcc@10.2.0/py-htseq@0.11.2.sh | 2 +-
.../gcc@10.2.0/spark@3.4.0.o22902088.exp-15-02 | 649 +++++++++++++++++++++
.../0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh | 76 +++
.../sdsc/expanse/0.17.3/gpu/b/yamls/modules.yaml | 1 +
4 files changed, 727 insertions(+), 1 deletion(-)
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.o22902088.exp-15-02
create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
[spack_cpu@exp-15-56 gcc@10.2.0]$ git stash pop
No stash entries found.
[spack_cpu@exp-15-56 gcc@10.2.0]$ less spark@3.4.0.sh
[spack_cpu@exp-15-56 gcc@10.2.0]$ sbatch spark@3.4.0.sh
Submitted batch job 23942019
[spack_cpu@exp-15-56 gcc@10.2.0]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
23942019 ind-share spark@3. spack_cp PD 0:00 1 (Priority)
23942013 ind-share bash spack_cp R 2:55 1 exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
23942013 ind-share bash spack_cp R 3:01 1 exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$
But don't forget any final changes that may be needed prior to running the build ...
[spack_cpu@exp-15-56 gcc@10.2.0]$ git diff
diff --git a/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
index 5234329413..d70c1f03ae 100644
--- a/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
+++ b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
@@ -2,7 +2,7 @@
#SBATCH --job-name=spark@3.4.0
#SBATCH --account=use300
-##SBATCH --reservation=rocky8u7_testing
+#SBATCH --reservation=root_73
#SBATCH --partition=ind-shared
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
[spack_cpu@exp-15-56 gcc@10.2.0]$ sbatch spark@3.4.0.sh
Submitted batch job 23942033
[spack_cpu@exp-15-56 gcc@10.2.0]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
23942033 ind-share spark@3. spack_cp R 0:04 1 exp-15-56
23942013 ind-share bash spack_cp R 5:53 1 exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$
Doh. Or that you also changed the Spack package for the application itself ...
==> Error: FetchError: Will not fetch spark@3.4.0
Add a checksum or use --no-checksum to skip this check.
/cm/shared/apps/spack/0.17.3/cpu/b/lib/spack/spack/package.py:1376, in do_fetch:
1373 .format(ck_msg))
1374
1375 if not ignore_checksum:
>> 1376 raise FetchError("Will not fetch %s" %
1377 self.spec.format('{name}{@version}'), ck_msg)
1378
1379 deprecated = spack.config.get('config:deprecated')
==> Error: Terminating after first install failure: FetchError: Will not fetch spark@3.4.0
Add a checksum or use --no-checksum to skip this check.
real 52.36
user 18.80
sys 8.50
ERROR: spack install failed.
To fix this issue, we'll need to walk through the pull request workflow again by placing the updated Spark package ...
[mkandes@login01 ~]$ cd .spack/0.17.3/cpu/b/var/spack/repos/mkandes/
[mkandes@login01 mkandes]$ ls
packages repo.yaml
[mkandes@login01 mkandes]$ cd packages/
[mkandes@login01 packages]$ ls
spark
[mkandes@login01 packages]$ cd spark/
[mkandes@login01 spark]$ ls
package.py __pycache__
[mkandes@login01 spark]$
[mkandes@login01 spark]$ grep 3.4.0 package.py
version('3.4.0', sha256='6c0366139dab03f9bfb788b27918e7e3d411adcfa5631de75ac91afac87fcf5f')
[mkandes@login01 spark]$
... into SDSC's custom Spack package repo ...
[mkandes@login02 gcc@10.2.0]$ cd /home/mkandes/software/spack/repos/mkandes/
[mkandes@login02 mkandes]$ ls
bin DEPLOYMENT.md LICENSE-MIT README.md
CHANGELOG.md etc NOTICE SECURITY.md
CONTRIBUTING.md lib pyproject.toml share
COPYRIGHT LICENSE-APACHE pytest.ini var
[mkandes@login02 mkandes]$ cd var/spack/repos/
builtin/ builtin.mock/ sdsc/ tutorial/
[mkandes@login02 mkandes]$ cd var/spack/repos/sdsc/packages/
[mkandes@login02 packages]$ ls
amber amdscalapack gdbm mvapich2 qchem trinity
amdblis aocc intel nwchem quantum-espresso vasp6
amdfftw aocl-sparse libbeagle openmpi relion vmd
amdlibflame gamess matlab orca star
amdlibm gaussian mbedtls perl tcl
[mkandes@login02 packages]$
Re-sync fork.
[mkandes@login02 packages]$ git fetch --tags upstream
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), 649 bytes | 10.00 KiB/s, done.
From https://github.com/sdsc/spack
c33518253b..e9bb2189ca sdsc-0.17.3 -> upstream/sdsc-0.17.3
[mkandes@login02 packages]$ git merge upstream/sdsc-0.17.3
Updating 90f124605b..e9bb2189ca
Fast-forward
[mkandes@login02 packages]$ git log
commit e9bb2189ca01894c673113228dd061cc046bc948 (HEAD -> sdsc-0.17.3, upstream/sdsc-0.17.3)
Merge: c33518253b 90f124605b
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jul 10 16:26:01 2023 -0700
Merge pull request #89 from mkandes/sdsc-0.17.3
Add spark@3.4.0 % gcc@10.2.0 to expanse/0.17.3/cpu/b
commit 90f124605b39fb79339dd2efe492ec1437b20b79 (origin/sdsc-0.17.3, origin/HEAD)
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jul 10 16:18:34 2023 -0700
Add spark@3.4.0 % gcc@10.2.0 to expanse/0.17.3/cpu/b
commit c33518253b5cd6096e34c63c6fea4a7cd6618c7b
Merge: e6d631d16f 51d103d923
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jun 5 14:08:54 2023 -0700
Merge pull request #83 from mkandes/sdsc-0.17.3-gh-81-yaml-modules-whitelist
[mkandes@login02 packages]$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com':
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 669 bytes | 669.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/mkandes/spack.git
90f124605b..e9bb2189ca sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login02 packages]$ git push --tags
Everything up-to-date
[mkandes@login02 packages]$
Copy custom Spack package to SDSC's Spack package repo in your fork.
[mkandes@login02 packages]$ cp -rp /home/mkandes/.spack/0.17.3/cpu/b/var/spack/repos/mkandes/packages/spark ./
[mkandes@login02 packages]$ cd spark/
[mkandes@login02 spark]$ rm __pycache__/
rm: cannot remove '__pycache__/': Is a directory
[mkandes@login02 spark]$ rm -rf __pycache__/
[mkandes@login02 spark]$ ls
package.py
[mkandes@login02 spark]$
[mkandes@login02 spark]$ cd ../
[mkandes@login02 packages]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
spark/
nothing added to commit but untracked files present (use "git add" to track)
[mkandes@login02 packages]$
[mkandes@login02 packages]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
spark/
nothing added to commit but untracked files present (use "git add" to track)
[mkandes@login02 packages]$ git add spark/
[mkandes@login02 packages]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
new file: spark/package.py
[mkandes@login02 packages]$ git commit
[sdsc-0.17.3 7e3d045306] Add custom spark package to sdsc package repo in sdsc-0.17.3
1 file changed, 64 insertions(+)
create mode 100644 var/spack/repos/sdsc/packages/spark/package.py
[mkandes@login02 packages]$ git log | head -n 10
commit 7e3d0453067e6d31c67922c47683148a9c5192e6
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jul 10 17:11:31 2023 -0700
Add custom spark package to sdsc package repo in sdsc-0.17.3
The custom change here is simply to include the latest version of
Spark at the time of this writing, which is Spark v3.4.0, and the sha256
hash associated with its downloadable tarball from the Spark project.
[mkandes@login02 packages]$ git push
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 64 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (9/9), 2.17 KiB | 2.17 MiB/s, done.
Total 9 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/mkandes/spack.git
e9bb2189ca..7e3d045306 sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login02 packages]$
Create pull request, review, then merge upstream into sdsc/spack repo.
Deploy custom Spack package to production instance from sdsc/spack repo ...
[spack_cpu@exp-15-56 gcc@10.2.0]$ git stash
Saved working directory and index state WIP on sdsc-0.17.3: e9bb2189ca Merge pull request #89 from mkandes/sdsc-0.17.3
[spack_cpu@exp-15-56 gcc@10.2.0]$ git pull
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 10 (delta 3), reused 9 (delta 3), pack-reused 0
Unpacking objects: 100% (10/10), 2.77 KiB | 404.00 KiB/s, done.
From https://github.com/sdsc/spack
e9bb2189ca..553657e0d4 sdsc-0.17.3 -> origin/sdsc-0.17.3
Updating e9bb2189ca..553657e0d4
Fast-forward
var/spack/repos/sdsc/packages/spark/package.py | 64 ++++++++++++++++++++++++++
1 file changed, 64 insertions(+)
create mode 100644 var/spack/repos/sdsc/packages/spark/package.py
[spack_cpu@exp-15-56 gcc@10.2.0]$ git stash pop
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: spark@3.4.0.sh
Untracked files:
(use "git add <file>..." to include in what will be committed)
../../../../../../../compilers.yaml
../../../../../../../licenses/aocc/
../../../../../../../licenses/intel/
../../../../../../../modules.yaml
../../../../../../../packages.yaml
spark@3.4.0.o23942033.exp-15-56
no changes added to commit (use "git add" and/or "git commit -a")
Dropped refs/stash@{0} (a10cca7b641ece23d8e0e39b6bbac6cf6ed640c6)
[spack_cpu@exp-15-56 gcc@10.2.0]$
Try spack build again ...
[spack_cpu@exp-15-56 gcc@10.2.0]$ sbatch spark@3.4.0.sh
Submitted batch job 23942559
[spack_cpu@exp-15-56 gcc@10.2.0]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
23942559 ind-share spark@3. spack_cp R 0:03 1 exp-15-56
23942013 ind-share bash spack_cp R 40:08 1 exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$
Check that build succeeded.
[spack_cpu@exp-15-56 gcc@10.2.0]$ ls -lahtr | tail -n 10
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 08:53 openblas@0.3.18-omp.sh
-rw-r--r-- 1 spack_cpu spack 2.2K Apr 21 08:54 openblas@0.3.18.sh
-rw-r--r-- 1 spack_cpu spack 27K Apr 21 08:56 openblas@0.3.18-i64.o21819638.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 09:00 openblas@0.3.18-i64.sh
-rw-r--r-- 1 spack_cpu spack 26K Jul 10 16:41 spark@3.4.0.o22902088.exp-15-02
-rw-r--r-- 1 spack_cpu spack 2.2K Jul 10 16:41 py-htseq@0.11.2.sh
-rw-r--r-- 1 spack_cpu spack 25K Jul 10 16:46 spark@3.4.0.o23942033.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.2K Jul 10 17:17 spark@3.4.0.sh
drwxr-sr-x 7 spack_cpu spack 268 Jul 10 17:19 .
-rw-r--r-- 1 spack_cpu spack 24K Jul 10 17:21 spark@3.4.0.o23942559.exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$ tail -n 10 spark@3.4.0.o23942559.exp-15-56
==> No patches needed for spark
==> spark: Executing phase: 'install'
==> spark: Successfully installed spark-3.4.0-sntnkcqkor2zmovkogcjf5qivzxdul5v
Fetch: 42.42s. Build: 2.84s. Total: 45.26s.
[+] /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/spark-3.4.0-sntnkcqkor2zmovkogcjf5qivzxdul5v
real 55.59
user 11.73
sys 2.01
==> Regenerating lmod module files
==> OpenFOAM bashrc env: /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/aocc-3.2.0/openfoam-2106-jz42us227mirxrhqjvojlaut2giuh74j/etc/bashrc
[spack_cpu@exp-15-56 gcc@10.2.0]$
[mkandes@login02 ~]$ spack find -lvdp spark
==> 2 installed packages
-- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
po6mvtn spark@3.4.0+hadoop /home/mkandes/.spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/spark-3.4.0-po6mvtnx2dirat72biironfv4ikcw5m2
bfyu354 hadoop@3.3.0 /home/mkandes/.spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/hadoop-3.3.0-bfyu354o4z5trshyxr6zkg463zdoat2f
27cv2ps openjdk@11.0.12_7 /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/openjdk-11.0.12_7-27cv2psb6xthjbwmjlx53otdbjxch6wg
sntnkcq spark@3.4.0+hadoop /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/spark-3.4.0-sntnkcqkor2zmovkogcjf5qivzxdul5v
bfyu354 hadoop@3.3.0 /home/mkandes/.spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/hadoop-3.3.0-bfyu354o4z5trshyxr6zkg463zdoat2f
27cv2ps openjdk@11.0.12_7 /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/openjdk-11.0.12_7-27cv2psb6xthjbwmjlx53otdbjxch6wg
[mkandes@login02 ~]$
Note, however, the module is not immediately visible for some reason ...
[mkandes_test@login02]~% module spider spark
----------------------------------------------------------------------------
spark:
----------------------------------------------------------------------------
Versions:
spark/2.4.8
spark/3.2.1
----------------------------------------------------------------------------
For detailed information about a specific "spark" package (including how to load the modules) use the module's full name. Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider spark/3.2.1
----------------------------------------------------------------------------
[mkandes_test@login02]~%
[mkandes@login02 ~]$ ls -lahtr /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/gcc/10.2.0/spark/3.4.0/
total 2.5K
drwxr-sr-x 3 spack_cpu spack 1 Jul 10 17:21 ..
drwxr-sr-x 2 spack_cpu spack 1 Jul 10 17:21 .
-rw-r--r-- 1 spack_cpu spack 2.4K Jul 10 17:32 sntnkcq.lua
[mkandes@login02 ~]$
It was discovered that both login nodes currently have a system spider cache that is preventing an updated view of the available modules ...
[mkandes@login02 ~]$ ls -lahtr /var/lib/lmod/mData/cacheDir
total 6.6M
-rw-r--r-- 1 root root 2.0M Jun 19 14:02 spiderT.old.lua
-rw-r--r-- 1 root root 1.4M Jun 19 14:02 spiderT.old.luac_5.3
-rw-r--r-- 1 root root 2.0M Jun 19 14:02 spiderT.lua
-rw-r--r-- 1 root root 1.4M Jun 19 14:02 spiderT.luac_5.3
drwxr-xr-x 2 root root 4.0K Jun 19 14:02 .
drwxr-xr-x 3 root root 4.0K Jun 19 14:02 ..
[mkandes@login02 ~]$
However, this can be ignored with the --ignore_cache
flag.
[spack_cpu@login02 ~]$ module --ignore_cache spider spark
----------------------------------------------------------------------------------------------------------------------------------------------------
spark:
----------------------------------------------------------------------------------------------------------------------------------------------------
Versions:
spark/2.4.8
spark/3.2.1
Other possible modules matches:
spark/3.4.0
----------------------------------------------------------------------------------------------------------------------------------------------------
To find other possible module matches execute:
$ module -r spider '.*spark.*'
----------------------------------------------------------------------------------------------------------------------------------------------------
For detailed information about a specific "spark" package (including how to load the modules) use the module's full name. Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider spark/3.2.1
----------------------------------------------------------------------------------------------------------------------------------------------------
[spack_cpu@login02 ~]$
To finalize this production deployment of Spark, we'll commit final changes to spec build script back to sdsc/repo, remove test build output and replace with final build output.
[spack_cpu@exp-15-56 gcc@10.2.0]$ git diff
diff --git a/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
index 5234329413..d70c1f03ae 100644
--- a/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
+++ b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
@@ -2,7 +2,7 @@
#SBATCH --job-name=spark@3.4.0
#SBATCH --account=use300
-##SBATCH --reservation=rocky8u7_testing
+#SBATCH --reservation=root_73
#SBATCH --partition=ind-shared
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
[spack_cpu@exp-15-56 gcc@10.2.0]$
[spack_cpu@exp-15-56 gcc@10.2.0]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: spark@3.4.0.sh
Untracked files:
(use "git add <file>..." to include in what will be committed)
../../../../../../../compilers.yaml
../../../../../../../licenses/aocc/
../../../../../../../licenses/intel/
../../../../../../../modules.yaml
../../../../../../../packages.yaml
spark@3.4.0.o23942033.exp-15-56
spark@3.4.0.o23942559.exp-15-56
no changes added to commit (use "git add" and/or "git commit -a")
[spack_cpu@exp-15-56 gcc@10.2.0]$
[spack_cpu@exp-15-56 gcc@10.2.0]$ ls -lahtr | tail -n 10
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 08:53 openblas@0.3.18-omp.sh
-rw-r--r-- 1 spack_cpu spack 2.2K Apr 21 08:54 openblas@0.3.18.sh
-rw-r--r-- 1 spack_cpu spack 27K Apr 21 08:56 openblas@0.3.18-i64.o21819638.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 09:00 openblas@0.3.18-i64.sh
-rw-r--r-- 1 spack_cpu spack 26K Jul 10 16:41 spark@3.4.0.o22902088.exp-15-02
-rw-r--r-- 1 spack_cpu spack 2.2K Jul 10 16:41 py-htseq@0.11.2.sh
-rw-r--r-- 1 spack_cpu spack 25K Jul 10 16:46 spark@3.4.0.o23942033.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.2K Jul 10 17:17 spark@3.4.0.sh
drwxr-sr-x 7 spack_cpu spack 268 Jul 10 17:19 .
-rw-r--r-- 1 spack_cpu spack 24K Jul 10 17:21 spark@3.4.0.o23942559.exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$
[spack_cpu@exp-15-56 gcc@10.2.0]$ git rm spark@3.4.0.o22902088.exp-15-02
rm 'etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.o22902088.exp-15-02'
[spack_cpu@exp-15-56 gcc@10.2.0]$ rm spark@3.4.0.o23942033.exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$ git add spark@3.4.0.sh
[spack_cpu@exp-15-56 gcc@10.2.0]$ git add spark@3.4.0.o23942559.exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
renamed: spark@3.4.0.o22902088.exp-15-02 -> spark@3.4.0.o23942559.exp-15-56
modified: spark@3.4.0.sh
Untracked files:
(use "git add <file>..." to include in what will be committed)
../../../../../../../compilers.yaml
../../../../../../../licenses/aocc/
../../../../../../../licenses/intel/
../../../../../../../modules.yaml
../../../../../../../packages.yaml
[spack_cpu@exp-15-56 gcc@10.2.0]$
[spack_cpu@exp-15-56 gcc@10.2.0]$ git commit
[sdsc-0.17.3 bd28762c3a] Deploy spark@3.4.0 % gcc@10.2.0 into prod within expanse/0.17.3/cpu/b
2 files changed, 93 insertions(+), 100 deletions(-)
rename etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/{spark@3.4.0.o22902088.exp-15-02 => spark@3.4.0.o23942559.exp-15-56} (72%)
[spack_cpu@exp-15-56 gcc@10.2.0]$
[spack_cpu@exp-15-56 gcc@10.2.0]$ git log
commit bd28762c3a74ea7b511b6c4abbedb3ab50b6ec48 (HEAD -> sdsc-0.17.3)
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Tue Jul 11 16:56:09 2023 -0700
Deploy spark@3.4.0 % gcc@10.2.0 into prod within expanse/0.17.3/cpu/b
commit 553657e0d4e92fb543587583665818cef36d4b94 (origin/sdsc-0.17.3, origin/HEAD)
Merge: e9bb2189ca 7e3d045306
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jul 10 17:15:28 2023 -0700
Merge pull request #90 from mkandes/sdsc-0.17.3
Add custom spark package to sdsc package repo in sdsc-0.17.3
commit 7e3d0453067e6d31c67922c47683148a9c5192e6
Author: Marty Kandes <mkandes@sdsc.edu>
Date: Mon Jul 10 17:11:31 2023 -0700
Add custom spark package to sdsc package repo in sdsc-0.17.3
The custom change here is simply to include the latest version of
[spack_cpu@exp-15-56 gcc@10.2.0]$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com':
Enumerating objects: 24, done.
Counting objects: 100% (24/24), done.
Delta compression using up to 128 threads
Compressing objects: 100% (9/9), done.
Writing objects: 100% (13/13), 8.50 KiB | 8.50 MiB/s, done.
Total 13 (delta 5), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (5/5), completed with 5 local objects.
To https://github.com/sdsc/spack.git
553657e0d4..bd28762c3a sdsc-0.17.3 -> sdsc-0.17.3
[spack_cpu@exp-15-56 gcc@10.2.0]$
Deployment complete. https://github.com/sdsc/spack/commit/bd28762c3a74ea7b511b6c4abbedb3ab50b6ec48
In general, what should follow next, if time permits and tests exist, is testing the newly deployed package in production before closing out the issue.
Using this issue as a first example workflow for deploying a new package into a production instance on Expanse. In this case, we will first deploy and test into a shared instance in my HOME directory on Expanse.
To setup the shared instance configuration, you first run the following configuration script ...