sdsc / spack

A flexible package manager that supports multiple versions, configurations, platforms, and compilers.
https://spack.io
Other
0 stars 4 forks source link

SDSC: PKG - expanse/0.17.3/cpu/b - Missing Spark (example application) #42

Closed nwolter closed 1 year ago

mkandes commented 1 year ago

Using this issue as a first example workflow for deploying a new package into a production instance on Expanse. In this case, we will first deploy and test into a shared instance in my HOME directory on Expanse.

To setup the shared instance configuration, you first run the following configuration script ...

[mkandes@login02 ~]$ cat shared-spack-instance.sh 
#!/usr/bin/env bash
#
# Load and configure shared Spack instance

declare -xr SHARED_SPACK_VERSION='0.17.3'
declare -xr SHARED_SPACK_INSTANCE_NAME='cpu'
declare -xr SHARED_SPACK_INSTANCE_VERSION='b'
declare -xr SHARED_SPACK_ROOT="/cm/shared/apps/spack/${SHARED_SPACK_VERSION}/${SHARED_SPACK_INSTANCE_NAME}/${SHARED_SPACK_INSTANCE_VERSION}"

declare -xr LOCAL_SPACK_NAMESPACE="${USER}"
declare -xr LOCAL_SPACK_TMPDIR='/tmp'
declare -xr LOCAL_SPACK_ROOT="${HOME}/.spack/${SHARED_SPACK_VERSION}/${SHARED_SPACK_INSTANCE_NAME}/${SHARED_SPACK_INSTANCE_VERSION}/${SHARED_SPACK_USER}"

module reset
module list
. "${SHARED_SPACK_ROOT}/share/spack/setup-env.sh"
printenv

mkdir -p "${LOCAL_SPACK_ROOT}"

mkdir -p "${LOCAL_SPACK_ROOT}/var/spack/repos/${LOCAL_SPACK_NAMESPACE}/packages"
tee -a "${LOCAL_SPACK_ROOT}/var/spack/repos/${LOCAL_SPACK_NAMESPACE}/repo.yaml" << EOF
repo:
  namespace: ${LOCAL_SPACK_NAMESPACE}
EOF

mkdir -p "${LOCAL_SPACK_ROOT}/var/spack/stage"
mkdir -p "${LOCAL_SPACK_ROOT}/var/spack/cache"
mkdir -p "${LOCAL_SPACK_ROOT}/share/spack/modules"
mkdir -p "${LOCAL_SPACK_ROOT}/share/spack/lmod"
mkdir -p "${LOCAL_SPACK_ROOT}/opt/spack"

architecture='${ARCHITECTURE}'
compilername='${COMPILERNAME}'
compilerver='${COMPILERVER}'
package='${PACKAGE}'
version='${VERSION}'
hash='${HASH}'

mkdir -p  "${LOCAL_SPACK_ROOT}/etc/spack"
tee -a "${LOCAL_SPACK_ROOT}/config.yaml" << EOF
config:
  install_tree: 
    root: ${LOCAL_SPACK_ROOT}/opt/spack
    projections:
      all: ${architecture}/${compilername}-${compilerver}/${package}-${version}-${hash}
  template_dirs:
    - ${SHARED_SPACK_ROOT}/share/spack/templates
  module_roots:
    tcl: ${LOCAL_SPACK_ROOT}/share/spack/modules
    lmod: ${LOCAL_SPACK_ROOT}/share/spack/lmod
  build_stage:
    - ${LOCAL_SPACK_ROOT}/var/spack/stage
    - ${LOCAL_SPACK_TMPDIR}/${USER}/spack-stage
  source_cache: ${LOCAL_SPACK_ROOT}/var/spack/cache
  misc_cache: ~/.spack/cache
  connect_timeout: 10
  verify_ssl: true
  suppress_gpg_warnings: false
  install_missing_compilers: false
  checksum: true
  dirty: false
  build_language: C
  locks: true
  build_jobs: 1
  ccache: false
  db_lock_timeout: 3
  package_lock_timeout: null
  shared_linking: 'rpath'
  allow_sgid: true
EOF

tee -a "${LOCAL_SPACK_ROOT}/repos.yaml" << EOF
repos:
  - ${LOCAL_SPACK_ROOT}/var/spack/repos/${LOCAL_SPACK_NAMESPACE}
EOF

tee -a "${LOCAL_SPACK_ROOT}/upstreams.yaml" << EOF
upstreams:
  spack-instance-1:
    install_tree: ${SHARED_SPACK_ROOT}/opt/spack
EOF
[mkandes@login02 ~]$
mkandes commented 1 year ago

Once the LOCAL_SPACK_ROOT is configured, you can then temporarily setup the shared instance with the following modifications in your '~/.bashrc` file.

[mkandes@login02 ~]$ cat .bashrc 
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
    PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/mkandes/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/mkandes/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/home/mkandes/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/mkandes/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

declare -xr SHARED_SPACK_VERSION='0.17.3'
declare -xr SHARED_SPACK_INSTANCE_NAME='cpu'
declare -xr SHARED_SPACK_INSTANCE_VERSION='b'
declare -xr SHARED_SPACK_ROOT="/cm/shared/apps/spack/${SHARED_SPACK_VERSION}/${SHARED_SPACK_INSTANCE_NAME}/${SHARED_SPACK_INSTANCE_VERSION}"

declare -xr LOCAL_SPACK_NAMESPACE="${USER}"
declare -xr LOCAL_SPACK_TMPDIR='/tmp'
declare -xr LOCAL_SPACK_ROOT="${HOME}/.spack/${SHARED_SPACK_VERSION}/${SHARED_SPACK_INSTANCE_NAME}/${SHARED_SPACK_INSTANCE_VERSION}/${SHARED_SPACK_USER}"

. "${SHARED_SPACK_ROOT}/share/spack/setup-env.sh"
module use "${LOCAL_SPACK_ROOT}/share/spack/lmod/linux-rocky8-x86_64"

alias spack="spack --config-scope ${LOCAL_SPACK_ROOT}"
[mkandes@login02 ~]$
mkandes commented 1 year ago

Upon starting a new shell session, you should have access to production Spack instance on Expanse, but it is now scoped to perform any installs into your LOCAL_SPACK_ROOT in ~/.spack.

[mkandes@login01 ~]$ spack --version
0.17.3-127-e6d631d16f
[mkandes@login01 ~]$ which spack
alias spack='spack --config-scope /home/mkandes/.spack/0.17.3/cpu/b/'
    spack ()
    { 
        : this is a shell function from: /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/setup-env.sh;
        : the real spack script is here: /cm/shared/apps/spack/0.17.3/cpu/b/bin/spack;
        _spack_shell_wrapper "$@";
        return $?
    }
[mkandes@login01 ~]$
mkandes commented 1 year ago

Before installing a spark via Spack, let's check if the version we want is available.

[mkandes@login01 ~]$ spack info spark
Package:   spark

Description:
    Apache Spark is a fast and general engine for large-scale data
    processing.

Homepage: https://spark.apache.org

Externally Detectable: 
    False

Tags: 
    None

Preferred version:  
    3.1.1    https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-without-hadoop.tgz

Safe versions:  
    3.1.1    https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-without-hadoop.tgz
    3.0.2    https://archive.apache.org/dist/spark/spark-3.0.2/spark-3.0.2-bin-without-hadoop.tgz
    3.0.1    https://archive.apache.org/dist/spark/spark-3.0.1/spark-3.0.1-bin-without-hadoop.tgz
    2.4.7    https://archive.apache.org/dist/spark/spark-2.4.7/spark-2.4.7-bin-without-hadoop.tgz
    2.3.0    https://archive.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-without-hadoop.tgz
    2.1.0    https://archive.apache.org/dist/spark/spark-2.1.0/spark-2.1.0-bin-without-hadoop.tgz
    2.0.2    https://archive.apache.org/dist/spark/spark-2.0.2/spark-2.0.2-bin-without-hadoop.tgz
    2.0.0    https://archive.apache.org/dist/spark/spark-2.0.0/spark-2.0.0-bin-without-hadoop.tgz
    1.6.2    https://archive.apache.org/dist/spark/spark-1.6.2/spark-1.6.2-bin-without-hadoop.tgz
    1.6.1    https://archive.apache.org/dist/spark/spark-1.6.1/spark-1.6.1-bin-without-hadoop.tgz
    1.6.0    https://archive.apache.org/dist/spark/spark-1.6.0/spark-1.6.0-bin-without-hadoop.tgz

Deprecated versions:  
    None

Variants:
    Name [Default]    When    Allowed values    Description
    ==============    ====    ==============    =================

    hadoop [off]      --      on, off           Build with Hadoop

Installation Phases:
    install

Build Dependencies:
    hadoop  java

Link Dependencies:
    None

Run Dependencies:
    hadoop  java

Virtual Packages: 
    None

[mkandes@login01 ~]$

The latest version at the time of this writing is: https://archive.apache.org/dist/spark/spark-3.4.0. As such, let's also plan to upgrade our spark Spack package to include the latest downloadable tarball.

mkandes commented 1 year ago

In this case, we'll likely want to install spark in the gcc@10.2.0 package dependency chain of expanse/0.17.3/cpu/b as this would allow us to more closely integrate it with either the python or R package environments we support there via Spack. Let's check what namespace and/or package repo the latest version of spark is currently coming from ...

[mkandes@login01 ~]$ spack spec -lNt --reuse spark@3.1.1 % gcc@10.2.0 ~hadoop
Input spec
--------------------------------
[    ]  .spark@3.1.1%gcc@10.2.0~hadoop

Concretized
--------------------------------
b6vwm6x  [    ]  builtin.spark@3.1.1%gcc@10.2.0 cflags="-O2 -march=native" cxxflags="-O2 -march=native" fflags="-O2 -march=native" ~hadoop arch=linux-rocky8-zen2
27cv2ps  [b r ]      ^builtin.openjdk@11.0.12_7%gcc@10.2.0 cflags="-O2 -march=native" cxxflags="-O2 -march=native" fflags="-O2 -march=native"  arch=linux-rocky8-zen2

[mkandes@login01 ~]$
mkandes commented 1 year ago

To upgrade the spark version, we'll need to modify the Preferred version in the package.py file to include the latest tarball and its hash. Note, however, before placing the new package in the sdsc spack package repo

[mkandes@login01 ~]$ ls $SHARED_SPACK_ROOT/var/spack/repos/sdsc/
packages  repo.yaml
[mkandes@login01 ~]$

we can test it locally in my personal repo setup as part of the shared spack instance configuration.

[mkandes@login01 ~]$ ls $LOCAL_SPACK_ROOT/var/spack/repos/mkandes/
packages  repo.yaml
[mkandes@login01 ~]$

Since the default package is currently coming from the builtin repo, let's go ahead and make a copy to edit and use for our test build.

[mkandes@login01 ~]$ cd $LOCAL_SPACK_ROOT/var/spack/repos/mkandes/packages
[mkandes@login01 packages]$ pwd
/home/mkandes/.spack/0.17.3/cpu/b/var/spack/repos/mkandes/packages
[mkandes@login01 packages]$ ls $SHARED_SPACK_ROOT/var/spack/repos/builtin/packages/spark
package.py  __pycache__
[mkandes@login01 packages]$ cp -rp $SHARED_SPACK_ROOT/var/spack/repos/builtin/packages/spark ./
[mkandes@login01 packages]$ pwd
/home/mkandes/.spack/0.17.3/cpu/b/var/spack/repos/mkandes/packages
[mkandes@login01 packages]$ ls
spark
[mkandes@login01 packages]$ cd spark/
[mkandes@login01 spark]$ rm __pycache__/
rm: cannot remove '__pycache__/': Is a directory
[mkandes@login01 spark]$ rm -rf __pycache__/
[mkandes@login01 spark]$ ls
package.py
[mkandes@login01 spark]$
mkandes commented 1 year ago

Download and compute sha256sum of the tarball.

mkandes@hardtack:~$ wget https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-without-hadoop.tgz
--2023-06-06 14:51:00--  https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-without-hadoop.tgz
Resolving archive.apache.org (archive.apache.org)... 65.108.204.189, 2a01:4f9:1a:a084::2
Connecting to archive.apache.org (archive.apache.org)|65.108.204.189|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 300271470 (286M) [application/x-gzip]
Saving to: ‘spark-3.4.0-bin-without-hadoop.tgz’

spark-3.4.0-bin-wit 100%[===================>] 286.36M  23.1MB/s    in 20s     

2023-06-06 14:51:21 (14.0 MB/s) - ‘spark-3.4.0-bin-without-hadoop.tgz’ saved [300271470/300271470]

mkandes@hardtack:~$ sha256sum spark-3.4.0-bin-without-hadoop.tgz 
6c0366139dab03f9bfb788b27918e7e3d411adcfa5631de75ac91afac87fcf5f  spark-3.4.0-bin-without-hadoop.tgz
mkandes@hardtack:~$
mkandes commented 1 year ago

And then add the new version to the package.py file.

version('3.4.0', sha256='6c0366139dab03f9bfb788b27918e7e3d411adcfa5631de75ac91afac87fcf5f')
mkandes commented 1 year ago

And now let's see if the new spark package from my personal package repo now has precedence ...

[mkandes@login01 ~]$ spack info spark
Package:   spark

Description:
    Apache Spark is a fast and general engine for large-scale data
    processing.

Homepage: https://spark.apache.org

Externally Detectable: 
    False

Tags: 
    None

Preferred version:  
    3.4.0    https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-without-hadoop.tgz

Safe versions:  
    3.4.0    https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-without-hadoop.tgz
    3.1.1    https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-without-hadoop.tgz
    3.0.2    https://archive.apache.org/dist/spark/spark-3.0.2/spark-3.0.2-bin-without-hadoop.tgz
    3.0.1    https://archive.apache.org/dist/spark/spark-3.0.1/spark-3.0.1-bin-without-hadoop.tgz
    2.4.7    https://archive.apache.org/dist/spark/spark-2.4.7/spark-2.4.7-bin-without-hadoop.tgz
    2.3.0    https://archive.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-without-hadoop.tgz
    2.1.0    https://archive.apache.org/dist/spark/spark-2.1.0/spark-2.1.0-bin-without-hadoop.tgz
    2.0.2    https://archive.apache.org/dist/spark/spark-2.0.2/spark-2.0.2-bin-without-hadoop.tgz
    2.0.0    https://archive.apache.org/dist/spark/spark-2.0.0/spark-2.0.0-bin-without-hadoop.tgz
    1.6.2    https://archive.apache.org/dist/spark/spark-1.6.2/spark-1.6.2-bin-without-hadoop.tgz
    1.6.1    https://archive.apache.org/dist/spark/spark-1.6.1/spark-1.6.1-bin-without-hadoop.tgz
    1.6.0    https://archive.apache.org/dist/spark/spark-1.6.0/spark-1.6.0-bin-without-hadoop.tgz

Deprecated versions:  
    None

Variants:
    Name [Default]    When    Allowed values    Description
    ==============    ====    ==============    =================

    hadoop [off]      --      on, off           Build with Hadoop

Installation Phases:
    install

Build Dependencies:
    hadoop  java

Link Dependencies:
    None

Run Dependencies:
    hadoop  java

Virtual Packages: 
    None

[mkandes@login01 ~]$
[mkandes@login01 ~]$ spack spec -lNt --reuse spark@3.4.0 % gcc@10.2.0 ~hadoop
Input spec
--------------------------------
[    ]  .spark@3.4.0%gcc@10.2.0~hadoop

Concretized
--------------------------------
pwgwwdj  [    ]  mkandes.spark@3.4.0%gcc@10.2.0 cflags="-O2 -march=native" cxxflags="-O2 -march=native" fflags="-O2 -march=native" ~hadoop arch=linux-rocky8-zen2
27cv2ps  [b r ]      ^builtin.openjdk@11.0.12_7%gcc@10.2.0 cflags="-O2 -march=native" cxxflags="-O2 -march=native" fflags="-O2 -march=native"  arch=linux-rocky8-zen2

[mkandes@login01 ~]$

Everything looks good. Now we need to construct a spec *.sh build script to deploy spark in my HOME directory and into the shared instance config. Once that build test is complete, we can submit the new Spack package for spark and the spec build script to deploy it into production as a pull request to the sdsc/spack repo.

mkandes commented 1 year ago
[mkandes@login01 ~]$ cp $SHARED_SPACK_ROOT/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/cmake@3.21.4.sh ./
[mkandes@login01 ~]$ mv cmake@3.21.4.sh spark@3.4.0.sh
[mkandes@login01 ~]$ vi spark@3.4.0.sh 
[mkandes@login01 ~]$ sbatch spark@3.4.0.sh 
Submitted batch job 22863818
[mkandes@login01 ~]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22863818 ind-share spark@3.  mkandes  R       0:03      1 exp-15-44
[mkandes@login01 ~]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22863818 ind-share spark@3.  mkandes  R       0:28      1 exp-15-44
[mkandes@login01 ~]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22863818 ind-share spark@3.  mkandes  R       0:51      1 exp-15-44
[mkandes@login01 ~]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22863818 ind-share spark@3.  mkandes  R       3:04      1 exp-15-44
[mkandes@login01 ~]$
mkandes commented 1 year ago

The following is the spark@3.4.0.sh spec build script submitted to Slurm on Expanse above to install Spark within my local Spack instance in my HOME directory prior to submitting script to the sdsc/spack repo as a pull request.

#!/usr/bin/env bash

#SBATCH --job-name=spark@3.4.0
#SBATCH --account=use300
##SBATCH --reservation=rocky8u7_testing
#SBATCH --partition=ind-shared
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G
#SBATCH --time=00:30:00
#SBATCH --output=%x.o%j.%N

declare -xr LOCAL_TIME="$(date +'%Y%m%dT%H%M%S%z')"
declare -xir UNIX_TIME="$(date +'%s')"

declare -xr LOCAL_SCRATCH_DIR="/scratch/${USER}/job_${SLURM_JOB_ID}"
declare -xr TMPDIR="${LOCAL_SCRATCH_DIR}"

declare -xr SYSTEM_NAME='expanse'

declare -xr SPACK_VERSION='0.17.3'
declare -xr SPACK_INSTANCE_NAME='cpu'
declare -xr SPACK_INSTANCE_VERSION='b'
declare -xr SPACK_INSTANCE_DIR="/cm/shared/apps/spack/${SPACK_VERSION}/${SPACK_INSTANCE_NAME}/${SPACK_INSTANCE_VERSION}"

declare -xr SLURM_JOB_SCRIPT="$(scontrol show job ${SLURM_JOB_ID} | awk -F= '/Command=/{print $2}')"
declare -xr SLURM_JOB_MD5SUM="$(md5sum ${SLURM_JOB_SCRIPT})"

declare -xr SCHEDULER_MODULE='slurm'

echo "${UNIX_TIME} ${SLURM_JOB_ID} ${SLURM_JOB_MD5SUM} ${SLURM_JOB_DEPENDENCY}" 
echo ""

cat "${SLURM_JOB_SCRIPT}"

module purge
module load "${SCHEDULER_MODULE}"
module list
. "${SPACK_INSTANCE_DIR}/share/spack/setup-env.sh"
shopt -s expand_aliases
source ~/.bashrc

declare -xr SPACK_PACKAGE='spark@3.4.0'
declare -xr SPACK_COMPILER='gcc@10.2.0'
declare -xr SPACK_VARIANTS='~hadoop'
declare -xr SPACK_DEPENDENCIES="^openjdk@1.8.0_265-b01/$(spack find --format '{hash:7}' openjdk@1.8.0_265-b01 % ${SPACK_COMPILER})"
declare -xr SPACK_SPEC="${SPACK_PACKAGE} % ${SPACK_COMPILER} ${SPACK_VARIANTS} ${SPACK_DEPENDENCIES}"

printenv

spack config get compilers
spack config get config
spack config get mirrors
spack config get modules
spack config get packages
spack config get repos
spack config get upstreams

time -p spack spec --long --namespaces --types --reuse "${SPACK_SPEC}"
if [[ "${?}" -ne 0 ]]; then
  echo 'ERROR: spack concretization failed.'
  exit 1
fi

time -p spack install --jobs "${SLURM_CPUS_PER_TASK}" --fail-fast --yes-to-all --reuse "${SPACK_SPEC}"
if [[ "${?}" -ne 0 ]]; then
  echo 'ERROR: spack install failed.'
  exit 1
fi

spack module lmod refresh -y

#sbatch --dependency="afterok:${SLURM_JOB_ID}" ''

sleep 30

Please note, the only custom changes needed to use the shared spack instance configuration setup in my ~/.bashrc file are these two lines ...

shopt -s expand_aliases
source ~/.bashrc

which can be easily removed prior to production deployment --- and even if forgotten prior to deployment, should have no effect.

mkandes commented 1 year ago
[mkandes@login01 ~]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
[mkandes@login01 ~]$ spack find -lvd spark@3.4.0
==> 1 installed package
-- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
pwgwwdj spark@3.4.0~hadoop
27cv2ps     openjdk@11.0.12_7

[mkandes@login01 ~]$
mkandes commented 1 year ago
[mkandes@login01 ~]$ module avail

---- /home/mkandes/.spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64 ----
   gcc/10.2.0/py-numpy/1.21.3/53ovhtc

- /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/Core -
   anaconda3/2021.05/q4munrg             matlab/2022b/lefe4oq
   aocc/3.2.0/io3s466                    mercurial/5.8/qmgrjvl
   aria2/1.35.0/q32jtg2                  parallel/20210922/sqru6rr
   entrezdirect/10.7.20190114/6pkkpx2    pigz/2.6/bgymyil
   gcc/10.2.0/npcyll4                    rclone/1.56.2/mldjorr
   gh/2.0.0/mkz3uxl                      sratoolkit/2.10.9/rn4humf
   git-lfs/2.11.0/kmruniy                subversion/1.14.0/qpzq6zs
   git/2.31.1/ldetm5y                    ucx/1.10.1/wla3unl
   intel/19.1.3.304/6pv46so
mkandes commented 1 year ago

Getting the module-based access working ...

[mkandes@login02 ~]$ spack module lmod refresh -y
==> Regenerating lmod module files
[mkandes@login02 ~]$ rm -rf .lmod.d/
[mkandes@login02 ~]$ module avail

---- /home/mkandes/.spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64 ----
   gcc/10.2.0/py-numpy/1.21.3/53ovhtc    gcc/10.2.0/spark/3.4.0/pwgwwdj

- /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/Core -
   anaconda3/2021.05/q4munrg             matlab/2022b/lefe4oq
   aocc/3.2.0/io3s466                    mercurial/5.8/qmgrjvl
   aria2/1.35.0/q32jtg2                  parallel/20210922/sqru6rr
   entrezdirect/10.7.20190114/6pkkpx2    pigz/2.6/bgymyil
   gcc/10.2.0/npcyll4                    rclone/1.56.2/mldjorr
   gh/2.0.0/mkz3uxl                      sratoolkit/2.10.9/rn4humf
   git-lfs/2.11.0/kmruniy                subversion/1.14.0/qpzq6zs
   git/2.31.1/ldetm5y                    ucx/1.10.1/wla3unl
   intel/19.1.3.304/6pv46so
mkandes commented 1 year ago

Spark is now available from my local Spack install_tree ..

[mkandes@login02 ~]$ module load gcc/10.2.0
[mkandes@login02 ~]$ module load gcc/10.2.0/spark/3.4.0
[mkandes@login02 ~]$ module list

Currently Loaded Modules:
  1) shared                      5) DefaultModules
  2) cpu/0.17.3b           (c)   6) gcc/10.2.0/npcyll4
  3) slurm/expanse/21.08.8       7) gcc/10.2.0/spark/3.4.0/pwgwwdj
  4) sdsc/1.0

  Where:
   c:  built natively for AMD Rome

[mkandes@login02 ~]$ which spark-submit
~/.spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/spark-3.4.0-pwgwwdjitbjz4gpj5nrckrc5sa75r2mn/bin/spark-submit
[mkandes@login02 ~]$
mkandes commented 1 year ago

Redo with openjdk@1.8.0 ... and moving to a local specs dir ...

[mkandes@login02 specs]$ sbatch spark@3.4.0.sh 
Submitted batch job 22900951
[mkandes@login02 specs]$ squeeu -u $USER
-bash: squeeu: command not found
[mkandes@login02 specs]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22900951 ind-share spark@3.  mkandes  R       0:55      1 exp-15-05
[mkandes@login02 specs]$ vi spark@3.4.0.sh 
[mkandes@login02 specs]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22900951 ind-share spark@3.  mkandes  R       2:42      1 exp-15-05
[mkandes@login02 specs]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
[mkandes@login02 specs]$ spack find -lvd spark@3.4.0
==> 1 installed package
-- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
rksnhif spark@3.4.0~hadoop
cgiyxcb     openjdk@1.8.0_265-b01

[mkandes@login02 specs]$ module avail

---- /home/mkandes/.spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64 ----
   gcc/10.2.0/py-numpy/1.21.3/53ovhtc    gcc/10.2.0/spark/3.4.0/rksnhif

- /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/Core -
   anaconda3/2021.05/q4munrg             matlab/2022b/lefe4oq
   aocc/3.2.0/io3s466                    mercurial/5.8/qmgrjvl
   aria2/1.35.0/q32jtg2                  parallel/20210922/sqru6rr
   entrezdirect/10.7.20190114/6pkkpx2    pigz/2.6/bgymyil
   gcc/10.2.0/npcyll4                    rclone/1.56.2/mldjorr
   gh/2.0.0/mkz3uxl                      sratoolkit/2.10.9/rn4humf
   git-lfs/2.11.0/kmruniy                subversion/1.14.0/qpzq6zs
   git/2.31.1/ldetm5y                    ucx/1.10.1/wla3unl
   intel/19.1.3.304/6pv46so

---------------------------- /cm/local/modulefiles -----------------------------
   cmjob        openmpi/mlnx/gcc/64/4.1.5a1        singularitypro/3.9
   lua/5.3.5    shared                      (L)    slurm/expanse/21.08.8 (L)

---------------------- /cm/shared/apps/access/modulefiles ----------------------
   accessusage/0.5-1    cue-login-env

---------------------------- /usr/share/modulefiles ----------------------------
[mkandes@login02 specs]$ ls
spark@3.4.0.o22900951.exp-15-05  spark@3.4.0.sh
[mkandes@login02 specs]$ pwd
/home/mkandes/.spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs
[mkandes@login02 specs]$
mkandes commented 1 year ago

Let's do this one more time ... forgot +hadoop is required, unless you configure spark to use an external version already running on your system.

[mkandes@login02 specs]$ spack find -lvd spark@3.4.0
==> 1 installed package
-- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
rksnhif spark@3.4.0~hadoop
cgiyxcb     openjdk@1.8.0_265-b01

[mkandes@login02 specs]$ spack uninstall spark@3.4.0
==> The following packages will be uninstalled:

    -- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
    rksnhif spark@3.4.0

==> Do you want to proceed? [y/N] y
==> Successfully uninstalled spark@3.4.0%gcc@10.2.0~hadoop arch=linux-rocky8-zen2/rksnhif
[mkandes@login02 specs]$
mkandes commented 1 year ago
[mkandes@login02 specs]$ cat spark@3.4.0.sh 
#!/usr/bin/env bash

#SBATCH --job-name=spark@3.4.0
#SBATCH --account=use300
##SBATCH --reservation=rocky8u7_testing
#SBATCH --partition=ind-shared
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G
#SBATCH --time=00:30:00
#SBATCH --output=%x.o%j.%N

declare -xr LOCAL_TIME="$(date +'%Y%m%dT%H%M%S%z')"
declare -xir UNIX_TIME="$(date +'%s')"

declare -xr LOCAL_SCRATCH_DIR="/scratch/${USER}/job_${SLURM_JOB_ID}"
declare -xr TMPDIR="${LOCAL_SCRATCH_DIR}"

declare -xr SYSTEM_NAME='expanse'

declare -xr SPACK_VERSION='0.17.3'
declare -xr SPACK_INSTANCE_NAME='cpu'
declare -xr SPACK_INSTANCE_VERSION='b'
declare -xr SPACK_INSTANCE_DIR="/cm/shared/apps/spack/${SPACK_VERSION}/${SPACK_INSTANCE_NAME}/${SPACK_INSTANCE_VERSION}"

declare -xr SLURM_JOB_SCRIPT="$(scontrol show job ${SLURM_JOB_ID} | awk -F= '/Command=/{print $2}')"
declare -xr SLURM_JOB_MD5SUM="$(md5sum ${SLURM_JOB_SCRIPT})"

declare -xr SCHEDULER_MODULE='slurm'

echo "${UNIX_TIME} ${SLURM_JOB_ID} ${SLURM_JOB_MD5SUM} ${SLURM_JOB_DEPENDENCY}" 
echo ""

cat "${SLURM_JOB_SCRIPT}"

module purge
module load "${SCHEDULER_MODULE}"
module list
. "${SPACK_INSTANCE_DIR}/share/spack/setup-env.sh"
shopt -s expand_aliases
source ~/.bashrc

declare -xr SPACK_PACKAGE='spark@3.4.0'
declare -xr SPACK_COMPILER='gcc@10.2.0'
declare -xr SPACK_VARIANTS='+hadoop'
declare -xr SPACK_DEPENDENCIES="^openjdk@11.0.12_7/$(spack find --format '{hash:7}' openjdk@11.0.12_7 % ${SPACK_COMPILER})"
declare -xr SPACK_SPEC="${SPACK_PACKAGE} % ${SPACK_COMPILER} ${SPACK_VARIANTS} ${SPACK_DEPENDENCIES}"

printenv

spack config get compilers
spack config get config
spack config get mirrors
spack config get modules
spack config get packages
spack config get repos
spack config get upstreams

time -p spack spec --long --namespaces --types --reuse "${SPACK_SPEC}"
if [[ "${?}" -ne 0 ]]; then
  echo 'ERROR: spack concretization failed.'
  exit 1
fi

time -p spack install --jobs "${SLURM_CPUS_PER_TASK}" --fail-fast --yes-to-all --reuse "${SPACK_SPEC}"
if [[ "${?}" -ne 0 ]]; then
  echo 'ERROR: spack install failed.'
  exit 1
fi

spack module lmod refresh -y

#sbatch --dependency="afterok:${SLURM_JOB_ID}" ''

sleep 30
mkandes commented 1 year ago
[mkandes@login02 specs]$ vi spark@3.4.0.sh 
[mkandes@login02 specs]$ rm spark@3.4.0.o22901977.exp-15-45
[mkandes@login02 specs]$ sbatch spark@3.4.0.sh 
Submitted batch job 22902088
[mkandes@login02 specs]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22902088 ind-share spark@3.  mkandes  R       7:35      1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22902088 ind-share spark@3.  mkandes  R      17:35      1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22902088 ind-share spark@3.  mkandes  R      19:11      1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22902088 ind-share spark@3.  mkandes  R      30:34      1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22902088 ind-share spark@3.  mkandes  R      40:21      1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          22902088 ind-share spark@3.  mkandes  R      41:38      1 exp-15-02
[mkandes@login02 specs]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
[mkandes@login02 specs]$ ls -lahtr
total 28K
drwxr-xr-x 3 mkandes use300    3 Jun  9 07:05 ..
-rw-r--r-- 1 mkandes use300 2.2K Jun  9 08:10 spark@3.4.0.sh
drwxr-xr-x 2 mkandes use300    4 Jun  9 08:10 .
-rw-r--r-- 1 mkandes use300  26K Jun  9 08:57 spark@3.4.0.o22902088.exp-15-02
[mkandes@login02 specs]$
mkandes commented 1 year ago

Picking back up where I left off last month ...

[mkandes@login01 ~]$ ls -lahtr /home/mkandes/.spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs
total 28K
drwxr-xr-x 3 mkandes use300    3 Jun  9 07:05 ..
-rw-r--r-- 1 mkandes use300 2.2K Jun  9 08:10 spark@3.4.0.sh
drwxr-xr-x 2 mkandes use300    4 Jun  9 08:10 .
-rw-r--r-- 1 mkandes use300  26K Jun  9 08:57 spark@3.4.0.o22902088.exp-15-02
[mkandes@login01 ~]$ module avail

---- /home/mkandes/.spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64 ----
   gcc/10.2.0/py-numpy/1.21.3/53ovhtc    gcc/10.2.0/spark/3.4.0/po6mvtn

- /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/Core -
   anaconda3/2021.05/q4munrg             matlab/2022b/lefe4oq
   aocc/3.2.0/io3s466                    mercurial/5.8/qmgrjvl
   aria2/1.35.0/q32jtg2                  parallel/20210922/sqru6rr
   entrezdirect/10.7.20190114/6pkkpx2    pigz/2.6/bgymyil
   gcc/10.2.0/npcyll4                    rclone/1.56.2/mldjorr
   gh/2.0.0/mkz3uxl                      sratoolkit/2.10.9/rn4humf
   git-lfs/2.11.0/kmruniy                subversion/1.14.0/qpzq6zs
   git/2.31.1/ldetm5y                    ucx/1.10.1/wla3unl
   intel/19.1.3.304/6pv46so

---------------------------- /cm/local/modulefiles -----------------------------
   cmjob        openmpi/mlnx/gcc/64/4.1.5a1        singularitypro/3.9
   lua/5.3.5    shared                      (L)    slurm/expanse/21.08.8 (L)

---------------------- /cm/shared/apps/access/modulefiles ----------------------
   accessusage/0.5-1    cue-login-env

---------------------------- /usr/share/modulefiles ----------------------------
[mkandes@login01 ~]$

... the next step would be to run tests against the locally deployed application before submitting it upstream as a pull request. Unfortunately, here in the case of Spark, the application testing is/was more complicated and is not working as intended and/or expected at this time. However, I will continue on and complete the deployment as an initial example for documenting the process, which will then be written up in the sdsc/spack project's documentation.

mkandes commented 1 year ago

Before updating your personal fork of the sdsc/spack repository with the spec build script, you'll need to first re-sync it with the sdsc/spack as its upstream. Using a command-line approach on Expanse, this workflow is as follows ...

[mkandes@login02 repos]$ git clone https://github.com/mkandes/spack.git
Cloning into 'spack'...
remote: Enumerating objects: 463470, done.
remote: Total 463470 (delta 0), reused 0 (delta 0), pack-reused 463470
Receiving objects: 100% (463470/463470), 370.43 MiB | 37.79 MiB/s, done.
Resolving deltas: 100% (203948/203948), done.
Updating files: 100% (10124/10124), done.
[mkandes@login02 repos]$ ls
spack
[mkandes@login02 repos]$ mv spack/ mkandes/
[mkandes@login02 repos]$ ls
mkandes
[mkandes@login02 repos]$ pwd
/home/mkandes/software/spack/repos
[mkandes@login02 repos]$ cd mkandes/
[mkandes@login02 mkandes]$ ls
bin              DEPLOYMENT.md   LICENSE-MIT     README.md
CHANGELOG.md     etc             NOTICE          SECURITY.md
CONTRIBUTING.md  lib             pyproject.toml  share
COPYRIGHT        LICENSE-APACHE  pytest.ini      var
[mkandes@login02 mkandes]$ git log
commit e6d631d16ff491cdbac076f05b0da1ded43590ff (HEAD -> sdsc-0.17.3, origin/sdsc-0.17.3, origin/HEAD)
Merge: 439fba6469 298234311b
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jun 5 11:37:43 2023 -0700

    Merge pull request #82 from mkandes/sdsc-0.17.3-yaml-modules-sdsc-gh-80

    Modify modules.yaml for expanse/0.17.3/cpu/b to whitelist py-jupyter-…

commit 298234311b4e2b75bec80b9335f6e538941100e4 (origin/sdsc-0.17.3-yaml-modules-sdsc-gh-80)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jun 5 11:21:52 2023 -0700

    Modify modules.yaml for expanse/0.17.3/cpu/b to whitelist py-jupyter-core

    This minor change allows us to explicitly generate an lmod modulefile
    for the py-jupyter-core Spack package that was installed implicitly as
    a dependency of py-jupyterlab.

    See https://github.com/sdsc/spack/issues/80

[mkandes@login02 mkandes]$ git fetch origin
[mkandes@login02 mkandes]$ git branch
* sdsc-0.17.3
[mkandes@login02 mkandes]$ git remote add upstream https://github.com/sdsc/spack.git
[mkandes@login02 mkandes]$ git fetch --tags upstream
remote: Enumerating objects: 1, done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 1
Unpacking objects: 100% (1/1), 687 bytes | 3.00 KiB/s, done.
From https://github.com/sdsc/spack
 * [new branch]            FASTMath/ATPESC18      -> upstream/FASTMath/ATPESC18
 * [new branch]            FASTMath/ATPESC19      -> upstream/FASTMath/ATPESC19
 * [new branch]            PR/openblas            -> upstream/PR/openblas
 * [new branch]            PR/pymock              -> upstream/PR/pymock
 * [new branch]            add-miniconda3@4.8.2-and-revert-4.7.12.1 -> upstream/add-miniconda3@4.8.2-and-revert-4.7.12.1
 * [new branch]            amklinv/amanzi         -> upstream/amklinv/amanzi
 * [new branch]            ascent                 -> upstream/ascent
 * [new branch]            autotools-cflags       -> upstream/autotools-cflags
 * [new branch]            barry/ceed             -> upstream/barry/ceed
 * [new branch]            boost-intel-toolset    -> upstream/boost-intel-toolset
 * [new branch]            bugfix/buildable-true-override-virtual -> upstream/bugfix/buildable-true-override-virtual
 * [new branch]            bugfix/bz2-libs        -> upstream/bugfix/bz2-libs
 * [new branch]            bugfix/compare-abstract-specs -> upstream/bugfix/compare-abstract-specs
 * [new branch]            bugfix/cray-no-module-preloaded -> upstream/bugfix/cray-no-module-preloaded
 * [new branch]            bugfix/env-activate-only-installed-concrete-specs -> upstream/bugfix/env-activate-only-installed-concrete-specs
 * [new branch]            bugfix/improved-uid-detection -> upstream/bugfix/improved-uid-detection
 * [new branch]            bugfix/m4_sierra_gcc_2 -> upstream/bugfix/m4_sierra_gcc_2
 * [new branch]            bugfix/multiarch-detection -> upstream/bugfix/multiarch-detection
 * [new branch]            bugfix/ordered-compiler-names -> upstream/bugfix/ordered-compiler-names
 * [new branch]            bugfix/sbang-filter-non-ascii-python-3 -> upstream/bugfix/sbang-filter-non-ascii-python-3
 * [new branch]            bugfix/setup-env       -> upstream/bugfix/setup-env
 * [new branch]            bugfix/shasta-hotfix-live-branch -> upstream/bugfix/shasta-hotfix-live-branch
 * [new branch]            bugfix/target-args-cray-compilers -> upstream/bugfix/target-args-cray-compilers
 * [new branch]            cdash/xsdk             -> upstream/cdash/xsdk
 * [new branch]            chain-doc-minor-improvement -> upstream/chain-doc-minor-improvement
 * [new branch]            concretizer            -> upstream/concretizer
 * [new branch]            cray-libsci-and-ver    -> upstream/cray-libsci-and-ver
 * [new branch]            dependabot/github_actions/actions/setup-python-4.6.0 -> upstream/dependabot/github_actions/actions/setup-python-4.6.0
 * [new branch]            dependabot/github_actions/docker/build-push-action-4.0.0 -> upstream/dependabot/github_actions/docker/build-push-action-4.0.0
 * [new branch]            dependabot/github_actions/docker/login-action-2.1.0 -> upstream/dependabot/github_actions/docker/login-action-2.1.0
 * [new branch]            dependabot/github_actions/docker/setup-buildx-action-2.5.0 -> upstream/dependabot/github_actions/docker/setup-buildx-action-2.5.0
 * [new branch]            dependabot/github_actions/docker/setup-qemu-action-2.1.0 -> upstream/dependabot/github_actions/docker/setup-qemu-action-2.1.0
 * [new branch]            develop                -> upstream/develop
 * [new branch]            docs/environment-scopes -> upstream/docs/environment-scopes
 * [new branch]            documentation/asp_based_solver -> upstream/documentation/asp_based_solver
 * [new branch]            eaf/200402-SpackEnvironmentDocs -> upstream/eaf/200402-SpackEnvironmentDocs
 * [new branch]            efischer/190121-SetupEnv -> upstream/efischer/190121-SetupEnv
 * [new branch]            efischer/190923-MultiPackagePrototype -> upstream/efischer/190923-MultiPackagePrototype
 * [new branch]            efischer/docs2         -> upstream/efischer/docs2
 * [new branch]            features/add-buildcache-list-to-database -> upstream/features/add-buildcache-list-to-database
 * [new branch]            features/anonymous_specs-fix -> upstream/features/anonymous_specs-fix
 * [new branch]            features/arch-swapping -> upstream/features/arch-swapping
 * [new branch]            features/cached-cmake-build-system -> upstream/features/cached-cmake-build-system
 * [new branch]            features/cflags-resuse -> upstream/features/cflags-resuse
 * [new branch]            features/cmake_deps    -> upstream/features/cmake_deps
 * [new branch]            features/compil        -> upstream/features/compil
 * [new branch]            features/detect-and-swap-current-python -> upstream/features/detect-and-swap-current-python
 * [new branch]            features/env-improvements-release -> upstream/features/env-improvements-release
 * [new branch]            features/env-pre-cmd   -> upstream/features/env-pre-cmd
 * [new branch]            features/find-linux-compilers-by-module -> upstream/features/find-linux-compilers-by-module
 * [new branch]            features/fix-cc-wrapper -> upstream/features/fix-cc-wrapper
 * [new branch]            features/improved-python-detection -> upstream/features/improved-python-detection
 * [new branch]            features/libs-command  -> upstream/features/libs-command
 * [new branch]            features/make-platform-command-line-option -> upstream/features/make-platform-command-line-option
 * [new branch]            features/more-coverage -> upstream/features/more-coverage
 * [new branch]            features/multi-download -> upstream/features/multi-download
 * [new branch]            features/parallel-mirror -> upstream/features/parallel-mirror
 * [new branch]            features/rstudio       -> upstream/features/rstudio
 * [new branch]            features/sandbox-builds -> upstream/features/sandbox-builds
 * [new branch]            features/setup_py      -> upstream/features/setup_py
 * [new branch]            features/shebang-warning-to-msg -> upstream/features/shebang-warning-to-msg
 * [new branch]            features/solver        -> upstream/features/solver
 * [new branch]            features/spack-mirror-verify -> upstream/features/spack-mirror-verify
 * [new branch]            features/spack-test-add-show-subcommand -> upstream/features/spack-test-add-show-subcommand
 * [new branch]            features/spack-test-hdf5-fix -> upstream/features/spack-test-hdf5-fix
 * [new branch]            features/spack-test-openmpi -> upstream/features/spack-test-openmpi
 * [new branch]            features/spack-test-openmpi-add-tests -> upstream/features/spack-test-openmpi-add-tests
 * [new branch]            features/stat_coral_test -> upstream/features/stat_coral_test
 * [new branch]            features/tty-output    -> upstream/features/tty-output
 * [new branch]            features/view-with-env -> upstream/features/view-with-env
 * [new branch]            features/virtual-packages -> upstream/features/virtual-packages
 * [new branch]            fish-shell-1           -> upstream/fish-shell-1
 * [new branch]            fix_cc                 -> upstream/fix_cc
 * [new branch]            flake8-sha256          -> upstream/flake8-sha256
 * [new branch]            fugaku-cross-compilation -> upstream/fugaku-cross-compilation
 * [new branch]            gh-pages               -> upstream/gh-pages
 * [new branch]            github/pr_templates    -> upstream/github/pr_templates
 * [new branch]            implicit-variables-setup -> upstream/implicit-variables-setup
 * [new branch]            legion-update          -> upstream/legion-update
 * [new branch]            libfabric-191-seems-to-work-on-darwin -> upstream/libfabric-191-seems-to-work-on-darwin
 * [new branch]            mfem                   -> upstream/mfem
 * [new branch]            mfem-updates           -> upstream/mfem-updates
 * [new branch]            mfu-cmake              -> upstream/mfu-cmake
 * [new branch]            mpbelhorn-aomp-3.9.0-wip-patch -> upstream/mpbelhorn-aomp-3.9.0-wip-patch
 * [new branch]            package/archer         -> upstream/package/archer
 * [new branch]            packages/gsl-external-cblas -> upstream/packages/gsl-external-cblas
 * [new branch]            packages/pruners-mpi   -> upstream/packages/pruners-mpi
 * [new branch]            packages/tulip         -> upstream/packages/tulip
 * [new branch]            parse-oneapi-version   -> upstream/parse-oneapi-version
 * [new branch]            pipelines-fix-develop-ambiguity -> upstream/pipelines-fix-develop-ambiguity
 * [new branch]            pipelines-improve-ci-job-label -> upstream/pipelines-improve-ci-job-label
 * [new branch]            proto/sys-packages-old -> upstream/proto/sys-packages-old
 * [new branch]            py-cffi-fix-checksum   -> upstream/py-cffi-fix-checksum
 * [new branch]            py26-bootstrap         -> upstream/py26-bootstrap
 * [new branch]            release_pipeline       -> upstream/release_pipeline
 * [new branch]            releases/v0.10.0       -> upstream/releases/v0.10.0
 * [new branch]            releases/v0.11.0       -> upstream/releases/v0.11.0
 * [new branch]            releases/v0.11.1       -> upstream/releases/v0.11.1
 * [new branch]            releases/v0.11.2       -> upstream/releases/v0.11.2
 * [new branch]            releases/v0.12         -> upstream/releases/v0.12
 * [new branch]            releases/v0.13         -> upstream/releases/v0.13
 * [new branch]            releases/v0.13-hwlloc-master -> upstream/releases/v0.13-hwlloc-master
 * [new branch]            releases/v0.14         -> upstream/releases/v0.14
 * [new branch]            releases/v0.15         -> upstream/releases/v0.15
 * [new branch]            releases/v0.16         -> upstream/releases/v0.16
 * [new branch]            retry_serial           -> upstream/retry_serial
 * [new branch]            revert-4.7.12.1        -> upstream/revert-4.7.12.1
 * [new branch]            sdsc-0.15.4            -> upstream/sdsc-0.15.4
 * [new branch]            sdsc-0.17.1            -> upstream/sdsc-0.17.1
 * [new branch]            sdsc-0.17.2            -> upstream/sdsc-0.17.2
 * [new branch]            sdsc-0.17.3            -> upstream/sdsc-0.17.3
 * [new branch]            sdsc-0.18.0            -> upstream/sdsc-0.18.0
 * [new branch]            sdsc-0.19.1            -> upstream/sdsc-0.19.1
 * [new branch]            separate-wrapper-decorator -> upstream/separate-wrapper-decorator
 * [new branch]            shorten-install-status-help -> upstream/shorten-install-status-help
 * [new branch]            silo-qt4               -> upstream/silo-qt4
 * [new branch]            spacktivate            -> upstream/spacktivate
 * [new branch]            spec-language-ideas    -> upstream/spec-language-ideas
 * [new branch]            spec-parse-inequalities -> upstream/spec-parse-inequalities
 * [new branch]            spec-target-flag-option -> upstream/spec-target-flag-option
 * [new branch]            stacks-debugging       -> upstream/stacks-debugging
 * [new branch]            test-gitlab-check-stays-pending -> upstream/test-gitlab-check-stays-pending
 * [new branch]            test/v0.13-with-fast-environments -> upstream/test/v0.13-with-fast-environments
 * [new branch]            testing/wip_xsdk       -> upstream/testing/wip_xsdk
 * [new branch]            tests/vtk-m-smoke-test-fix -> upstream/tests/vtk-m-smoke-test-fix
 * [new branch]            topic-libSplash160     -> upstream/topic-libSplash160
 * [new branch]            tutorials/advanced_packaging -> upstream/tutorials/advanced_packaging
 * [new branch]            update/ipopt           -> upstream/update/ipopt
 * [new branch]            v-dobrev/extend-default-spec-queries -> upstream/v-dobrev/extend-default-spec-queries
 * [new branch]            vendor-clingo          -> upstream/vendor-clingo
 * [new branch]            z3-bootstrap-2.6       -> upstream/z3-bootstrap-2.6
 * [new branch]            z3-concretizer         -> upstream/z3-concretizer
[mkandes@login02 mkandes]$ git merge upstream/sdsc-0.17.3
Updating e6d631d16f..c33518253b
Fast-forward
 etc/spack/sdsc/expanse/0.17.3/gpu/b/yamls/modules.yaml | 1 +
 1 file changed, 1 insertion(+)
[mkandes@login02 mkandes]$ git log
commit c33518253b5cd6096e34c63c6fea4a7cd6618c7b (HEAD -> sdsc-0.17.3, upstream/sdsc-0.17.3)
Merge: e6d631d16f 51d103d923
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jun 5 14:08:54 2023 -0700

    Merge pull request #83 from mkandes/sdsc-0.17.3-gh-81-yaml-modules-whitelist

    Modify modules.yaml for expanse/0.17.3/gpu/b to whitelist py-jupyter-…

commit 51d103d923b635f0b7908b4c8a3fa69815e9a655 (origin/sdsc-0.17.3-gh-81-yaml-modules-whitelist)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jun 5 14:07:21 2023 -0700

    Modify modules.yaml for expanse/0.17.3/gpu/b to whitelist py-jupyter-core

    See https://github.com/sdsc/spack/issues/81

commit e6d631d16ff491cdbac076f05b0da1ded43590ff (origin/sdsc-0.17.3, origin/HEAD)
Merge: 439fba6469 298234311b
Author: Marty Kandes <mkandes@sdsc.edu>
[mkandes@login02 mkandes]$ ls
bin              DEPLOYMENT.md   LICENSE-MIT     README.md
CHANGELOG.md     etc             NOTICE          SECURITY.md
CONTRIBUTING.md  lib             pyproject.toml  share
COPYRIGHT        LICENSE-APACHE  pytest.ini      var
[mkandes@login02 mkandes]$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com': 
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 707 bytes | 707.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/mkandes/spack.git
   e6d631d16f..c33518253b  sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login02 mkandes]$ git push --tags
Everything up-to-date
[mkandes@login02 mkandes]$

We will also provide in the future a workflow using the GitHub web interface as well.

mkandes commented 1 year ago

Next, copy both the new spec build script and test build output files to the correct location in your fork of the sdsc/spack repository ...

[mkandes@login02 specs]$ pwd
/home/mkandes/.spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs
[mkandes@login02 specs]$ ls
spark@3.4.0.o22902088.exp-15-02  spark@3.4.0.sh
[mkandes@login02 specs]$
[mkandes@login02 gcc@10.2.0]$ pwd
/home/mkandes/software/spack/repos/mkandes/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0
[mkandes@login02 gcc@10.2.0]$ cp -rp /home/mkandes/.spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/spark@3.4.0.sh ./
[mkandes@login02 gcc@10.2.0]$ cp -rp /home/mkandes/.spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/spark@3.4.0.o22902088.exp-15-02 ./
[mkandes@login02 gcc@10.2.0]$ ls | grep spark
spark@3.4.0.o22902088.exp-15-02
spark@3.4.0.sh
[mkandes@login02 gcc@10.2.0]$

Once the new spec build script and the test build output file are placed in the correct location, add the build script to the end of existing package dependency chain for the production deployment.

[mkandes@login02 gcc@10.2.0]$ vi py-htseq@0.11.2.sh 
[mkandes@login02 gcc@10.2.0]$ grep spark py-htseq@0.11.2.sh 
sbatch --dependency="afterok:${SLURM_JOB_ID}" 'spark@3.4.0.sh'
[mkandes@login02 gcc@10.2.0]$

See https://github.com/mkandes/spack/blob/sdsc-0.17.3/etc/spack/sdsc/expanse/0.17.3/cpu/b/SPECS.md

OR

... check the deployed instance itself for the last known set of deployed build scripts to find the appropriate location to link in the new build script into the dependency chain ...

[mkandes@login02 gcc@10.2.0]$ pwd
/cm/shared/apps/spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0
[mkandes@login02 gcc@10.2.0]$ ls -lahtr | tail -n 30
-rw-r--r-- 1 spack_cpu spack  32K Apr 16 00:32 jellyfish@2.2.7.o21704016.exp-15-56
-rw-r--r-- 1 spack_cpu spack  42K Apr 16 00:36 salmon@1.4.0.o21704017.exp-15-56
-rw-r--r-- 1 spack_cpu spack  36K Apr 16 00:39 iq-tree@2.1.3.o21704079.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.3K Apr 16 10:11 iq-tree@2.1.3.sh
-rw-r--r-- 1 spack_cpu spack  22K Apr 16 10:17 hisat2@2.2.0.o21711278.exp-15-56
-rw-r--r-- 1 spack_cpu spack  91K Apr 16 10:20 perl-bioperl@1.7.6.o21711315.exp-15-56
-rw-r--r-- 1 spack_cpu spack  57K Apr 16 10:21 py-pysam@0.15.3.o21711330.exp-15-56
-rw-r--r-- 1 spack_cpu spack  30K Apr 16 11:54 ucx@1.10.1.o21712584.exp-15-56
-rw-r--r-- 1 spack_cpu spack  34K Apr 16 12:07 openmpi@4.1.3.o21712606.exp-15-56
-rw-r--r-- 1 spack_cpu spack 3.1K Apr 17 07:02 openmpi@4.1.1.sh
-rw-r--r-- 1 spack_cpu spack 3.1K Apr 17 07:02 openmpi@4.1.3.sh
-rw-r--r-- 1 spack_cpu spack  32K Apr 17 07:16 openmpi@4.1.1.o21730146.exp-15-56
-rw-r--r-- 1 spack_cpu spack  36K Apr 17 07:16 openmpi@3.1.6.o21730154.exp-15-56
drwxr-sr-x 2 spack_cpu spack  115 Apr 17 07:30 openmpi@4.1.3
-rw-r--r-- 1 spack_cpu spack 2.5K Apr 17 07:47 mvapich2@2.3.7.sh
-rw-r--r-- 1 spack_cpu spack 2.0K Apr 17 07:48 intel-mpi@2019.10.317.sh
-rw-r--r-- 1 spack_cpu spack  35K Apr 17 07:58 openmpi@3.1.6.o21730681.exp-15-56
-rw-r--r-- 1 spack_cpu spack  29K Apr 17 08:00 mvapich2@2.3.7.o21730613.exp-15-56
-rw-r--r-- 1 spack_cpu spack 3.5K Apr 17 08:07 openmpi@3.1.6.sh
-rw-r--r-- 1 spack_cpu spack  35K Apr 17 08:17 openmpi@3.1.6.o21730823.exp-15-56
-rw-r--r-- 1 spack_cpu spack  22K Apr 17 18:44 intel-mpi@2019.10.317.o21739793.exp-15-56
drwxr-sr-x 2 spack_cpu spack   47 Apr 17 19:09 mvapich2@2.3.7
drwxr-sr-x 2 spack_cpu spack   45 Apr 17 19:43 intel-mpi@2019.10.317
drwxr-sr-x 7 spack_cpu spack   43 Apr 20 08:23 ..
drwxr-sr-x 2 spack_cpu spack    1 Apr 21 08:42 openmpi@4.1.1
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 08:53 openblas@0.3.18-omp.sh
-rw-r--r-- 1 spack_cpu spack 2.2K Apr 21 08:54 openblas@0.3.18.sh
-rw-r--r-- 1 spack_cpu spack  27K Apr 21 08:56 openblas@0.3.18-i64.o21819638.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 09:00 openblas@0.3.18-i64.sh
drwxr-sr-x 7 spack_cpu spack  264 Apr 21 09:00 .
[mkandes@login02 gcc@10.2.0]$

We are working on some additional tooling to make this discovery process simpler in the future. For now, best effort to keep things organized is what we aim to do. Using the SPECS.md documentation on the dependency chains for this example.

mkandes commented 1 year ago

When ready, commit the changes to your fork. Please note, however, in this case this is a direct commit on the sdsc-0.17.3 deployment branch. In general, you should first created a spec branch off of the deployment branch before adding and committing these changes to the fork.

[mkandes@login02 gcc@10.2.0]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   py-htseq@0.11.2.sh

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    spark@3.4.0.o22902088.exp-15-02
    spark@3.4.0.sh

It took 3.10 seconds to enumerate untracked files. 'status -uno'
may speed it up, but you have to be careful not to forget to add
new files yourself (see 'git help status').
no changes added to commit (use "git add" and/or "git commit -a")
[mkandes@login02 gcc@10.2.0]$ git add -u
[mkandes@login02 gcc@10.2.0]$ git add spark@3.4.0.o22902088.exp-15-02
[mkandes@login02 gcc@10.2.0]$ git add spark@3.4.0.sh
[mkandes@login02 gcc@10.2.0]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    modified:   py-htseq@0.11.2.sh
    new file:   spark@3.4.0.o22902088.exp-15-02
    new file:   spark@3.4.0.sh

[mkandes@login02 gcc@10.2.0]$ git commit
[sdsc-0.17.3 90f124605b] Add spark@3.4.0 % gcc@10.2.0 to expanse/0.17.3/cpu/b
 3 files changed, 726 insertions(+), 1 deletion(-)
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.o22902088.exp-15-02
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
[mkandes@login02 gcc@10.2.0]$ git log
commit 90f124605b39fb79339dd2efe492ec1437b20b79 (HEAD -> sdsc-0.17.3)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jul 10 16:18:34 2023 -0700

    Add spark@3.4.0 % gcc@10.2.0 to expanse/0.17.3/cpu/b
mkandes commented 1 year ago

Push the changes back to your fork.

[mkandes@login02 gcc@10.2.0]$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com': 
Enumerating objects: 25, done.
Counting objects: 100% (25/25), done.
Delta compression using up to 64 threads
Compressing objects: 100% (10/10), done.
Writing objects: 100% (14/14), 9.04 KiB | 4.52 MiB/s, done.
Total 14 (delta 6), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (6/6), completed with 5 local objects.
To https://github.com/mkandes/spack.git
   c33518253b..90f124605b  sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login02 gcc@10.2.0]$
mkandes commented 1 year ago

Create pull request, review pull request (with other team members), merge pull request with sdsc/spack deployment branch prior to production deployment.

https://github.com/sdsc/spack/pull/89

mkandes commented 1 year ago

When ready to deploy package into production, login to role account, start interactive session for reserved node, pull down changes from sdsc/spack repo and deploy ...

[spack_cpu@login02 ~]$ srun --partition=ind-shared --reservation=root_73  --account=use300 --nodes=1 --nodelist=exp-15-56 --ntasks-per-node=1 --cpus-per-task=16 --mem=32G --time=12:00:00 --pty --wait=0 /bin/bash
[spack_cpu@exp-15-56 ~]$ cd /cm/shared/apps/spack/0.17.3/cpu/b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/
[spack_cpu@exp-15-56 gcc@10.2.0]$ git stash
No local changes to save
[spack_cpu@exp-15-56 gcc@10.2.0]$ git pull
remote: Enumerating objects: 27, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 27 (delta 9), reused 15 (delta 8), pack-reused 9
Unpacking objects: 100% (27/27), 11.38 KiB | 315.00 KiB/s, done.
From https://github.com/sdsc/spack
   e6d631d16f..e9bb2189ca  sdsc-0.17.3 -> origin/sdsc-0.17.3
Updating e6d631d16f..e9bb2189ca
Fast-forward
 .../cpu/b/specs/gcc@10.2.0/py-htseq@0.11.2.sh      |   2 +-
 .../gcc@10.2.0/spark@3.4.0.o22902088.exp-15-02     | 649 +++++++++++++++++++++
 .../0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh   |  76 +++
 .../sdsc/expanse/0.17.3/gpu/b/yamls/modules.yaml   |   1 +
 4 files changed, 727 insertions(+), 1 deletion(-)
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.o22902088.exp-15-02
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
[spack_cpu@exp-15-56 gcc@10.2.0]$ git stash pop
No stash entries found.
[spack_cpu@exp-15-56 gcc@10.2.0]$ less spark@3.4.0.sh 
[spack_cpu@exp-15-56 gcc@10.2.0]$ sbatch spark@3.4.0.sh 
Submitted batch job 23942019
[spack_cpu@exp-15-56 gcc@10.2.0]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          23942019 ind-share spark@3. spack_cp PD       0:00      1 (Priority)
          23942013 ind-share     bash spack_cp  R       2:55      1 exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          23942013 ind-share     bash spack_cp  R       3:01      1 exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$
mkandes commented 1 year ago

But don't forget any final changes that may be needed prior to running the build ...

[spack_cpu@exp-15-56 gcc@10.2.0]$ git diff
diff --git a/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
index 5234329413..d70c1f03ae 100644
--- a/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
+++ b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
@@ -2,7 +2,7 @@

 #SBATCH --job-name=spark@3.4.0
 #SBATCH --account=use300
-##SBATCH --reservation=rocky8u7_testing
+#SBATCH --reservation=root_73
 #SBATCH --partition=ind-shared
 #SBATCH --nodes=1
 #SBATCH --ntasks-per-node=1
[spack_cpu@exp-15-56 gcc@10.2.0]$ sbatch spark@3.4.0.sh 
Submitted batch job 23942033
[spack_cpu@exp-15-56 gcc@10.2.0]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          23942033 ind-share spark@3. spack_cp  R       0:04      1 exp-15-56
          23942013 ind-share     bash spack_cp  R       5:53      1 exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$
mkandes commented 1 year ago

Doh. Or that you also changed the Spack package for the application itself ...

==> Error: FetchError: Will not fetch spark@3.4.0
    Add a checksum or use --no-checksum to skip this check.

/cm/shared/apps/spack/0.17.3/cpu/b/lib/spack/spack/package.py:1376, in do_fetch:
       1373                              .format(ck_msg))
       1374
       1375            if not ignore_checksum:
  >>   1376                raise FetchError("Will not fetch %s" %
       1377                                 self.spec.format('{name}{@version}'), ck_msg)
       1378
       1379        deprecated = spack.config.get('config:deprecated')

==> Error: Terminating after first install failure: FetchError: Will not fetch spark@3.4.0
    Add a checksum or use --no-checksum to skip this check.
real 52.36
user 18.80
sys 8.50
ERROR: spack install failed.
mkandes commented 1 year ago

To fix this issue, we'll need to walk through the pull request workflow again by placing the updated Spark package ...

[mkandes@login01 ~]$ cd .spack/0.17.3/cpu/b/var/spack/repos/mkandes/
[mkandes@login01 mkandes]$ ls
packages  repo.yaml
[mkandes@login01 mkandes]$ cd packages/
[mkandes@login01 packages]$ ls
spark
[mkandes@login01 packages]$ cd spark/
[mkandes@login01 spark]$ ls
package.py  __pycache__
[mkandes@login01 spark]$
[mkandes@login01 spark]$ grep 3.4.0 package.py 
    version('3.4.0', sha256='6c0366139dab03f9bfb788b27918e7e3d411adcfa5631de75ac91afac87fcf5f')
[mkandes@login01 spark]$

... into SDSC's custom Spack package repo ...

[mkandes@login02 gcc@10.2.0]$ cd /home/mkandes/software/spack/repos/mkandes/
[mkandes@login02 mkandes]$ ls
bin              DEPLOYMENT.md   LICENSE-MIT     README.md
CHANGELOG.md     etc             NOTICE          SECURITY.md
CONTRIBUTING.md  lib             pyproject.toml  share
COPYRIGHT        LICENSE-APACHE  pytest.ini      var
[mkandes@login02 mkandes]$ cd var/spack/repos/
builtin/      builtin.mock/ sdsc/         tutorial/     
[mkandes@login02 mkandes]$ cd var/spack/repos/sdsc/packages/
[mkandes@login02 packages]$ ls
amber        amdscalapack  gdbm       mvapich2  qchem             trinity
amdblis      aocc          intel      nwchem    quantum-espresso  vasp6
amdfftw      aocl-sparse   libbeagle  openmpi   relion            vmd
amdlibflame  gamess        matlab     orca      star
amdlibm      gaussian      mbedtls    perl      tcl
[mkandes@login02 packages]$
mkandes commented 1 year ago

Re-sync fork.

[mkandes@login02 packages]$ git fetch --tags upstream
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), 649 bytes | 10.00 KiB/s, done.
From https://github.com/sdsc/spack
   c33518253b..e9bb2189ca  sdsc-0.17.3 -> upstream/sdsc-0.17.3
[mkandes@login02 packages]$ git merge upstream/sdsc-0.17.3
Updating 90f124605b..e9bb2189ca
Fast-forward
[mkandes@login02 packages]$ git log
commit e9bb2189ca01894c673113228dd061cc046bc948 (HEAD -> sdsc-0.17.3, upstream/sdsc-0.17.3)
Merge: c33518253b 90f124605b
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jul 10 16:26:01 2023 -0700

    Merge pull request #89 from mkandes/sdsc-0.17.3

    Add spark@3.4.0 % gcc@10.2.0 to expanse/0.17.3/cpu/b

commit 90f124605b39fb79339dd2efe492ec1437b20b79 (origin/sdsc-0.17.3, origin/HEAD)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jul 10 16:18:34 2023 -0700

    Add spark@3.4.0 % gcc@10.2.0 to expanse/0.17.3/cpu/b

commit c33518253b5cd6096e34c63c6fea4a7cd6618c7b
Merge: e6d631d16f 51d103d923
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jun 5 14:08:54 2023 -0700

    Merge pull request #83 from mkandes/sdsc-0.17.3-gh-81-yaml-modules-whitelist
[mkandes@login02 packages]$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com': 
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 669 bytes | 669.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/mkandes/spack.git
   90f124605b..e9bb2189ca  sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login02 packages]$ git push --tags
Everything up-to-date
[mkandes@login02 packages]$
mkandes commented 1 year ago

Copy custom Spack package to SDSC's Spack package repo in your fork.

[mkandes@login02 packages]$ cp -rp /home/mkandes/.spack/0.17.3/cpu/b/var/spack/repos/mkandes/packages/spark ./
[mkandes@login02 packages]$ cd spark/
[mkandes@login02 spark]$ rm __pycache__/
rm: cannot remove '__pycache__/': Is a directory
[mkandes@login02 spark]$ rm -rf __pycache__/
[mkandes@login02 spark]$ ls
package.py
[mkandes@login02 spark]$
mkandes commented 1 year ago
[mkandes@login02 spark]$ cd ../
[mkandes@login02 packages]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    spark/

nothing added to commit but untracked files present (use "git add" to track)
[mkandes@login02 packages]$
mkandes commented 1 year ago
[mkandes@login02 packages]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    spark/

nothing added to commit but untracked files present (use "git add" to track)
[mkandes@login02 packages]$ git add spark/
[mkandes@login02 packages]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    new file:   spark/package.py

[mkandes@login02 packages]$ git commit
[sdsc-0.17.3 7e3d045306] Add custom spark package to sdsc package repo in sdsc-0.17.3
 1 file changed, 64 insertions(+)
 create mode 100644 var/spack/repos/sdsc/packages/spark/package.py
[mkandes@login02 packages]$ git log | head -n 10
commit 7e3d0453067e6d31c67922c47683148a9c5192e6
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jul 10 17:11:31 2023 -0700

    Add custom spark package to sdsc package repo in sdsc-0.17.3

    The custom change here is simply to include the latest version of
    Spark at the time of this writing, which is Spark v3.4.0, and the sha256
    hash associated with its downloadable tarball from the Spark project.

[mkandes@login02 packages]$ git push
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 64 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (9/9), 2.17 KiB | 2.17 MiB/s, done.
Total 9 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/mkandes/spack.git
   e9bb2189ca..7e3d045306  sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login02 packages]$
mkandes commented 1 year ago

Create pull request, review, then merge upstream into sdsc/spack repo.

https://github.com/sdsc/spack/pull/90

mkandes commented 1 year ago

Deploy custom Spack package to production instance from sdsc/spack repo ...

[spack_cpu@exp-15-56 gcc@10.2.0]$ git stash
Saved working directory and index state WIP on sdsc-0.17.3: e9bb2189ca Merge pull request #89 from mkandes/sdsc-0.17.3
[spack_cpu@exp-15-56 gcc@10.2.0]$ git pull
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 10 (delta 3), reused 9 (delta 3), pack-reused 0
Unpacking objects: 100% (10/10), 2.77 KiB | 404.00 KiB/s, done.
From https://github.com/sdsc/spack
   e9bb2189ca..553657e0d4  sdsc-0.17.3 -> origin/sdsc-0.17.3
Updating e9bb2189ca..553657e0d4
Fast-forward
 var/spack/repos/sdsc/packages/spark/package.py | 64 ++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 var/spack/repos/sdsc/packages/spark/package.py
[spack_cpu@exp-15-56 gcc@10.2.0]$ git stash pop
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   spark@3.4.0.sh

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    ../../../../../../../compilers.yaml
    ../../../../../../../licenses/aocc/
    ../../../../../../../licenses/intel/
    ../../../../../../../modules.yaml
    ../../../../../../../packages.yaml
    spark@3.4.0.o23942033.exp-15-56

no changes added to commit (use "git add" and/or "git commit -a")
Dropped refs/stash@{0} (a10cca7b641ece23d8e0e39b6bbac6cf6ed640c6)
[spack_cpu@exp-15-56 gcc@10.2.0]$
mkandes commented 1 year ago

Try spack build again ...

[spack_cpu@exp-15-56 gcc@10.2.0]$ sbatch spark@3.4.0.sh 
Submitted batch job 23942559
[spack_cpu@exp-15-56 gcc@10.2.0]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          23942559 ind-share spark@3. spack_cp  R       0:03      1 exp-15-56
          23942013 ind-share     bash spack_cp  R      40:08      1 exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$
mkandes commented 1 year ago

Check that build succeeded.

[spack_cpu@exp-15-56 gcc@10.2.0]$ ls -lahtr | tail -n 10
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 08:53 openblas@0.3.18-omp.sh
-rw-r--r-- 1 spack_cpu spack 2.2K Apr 21 08:54 openblas@0.3.18.sh
-rw-r--r-- 1 spack_cpu spack  27K Apr 21 08:56 openblas@0.3.18-i64.o21819638.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 09:00 openblas@0.3.18-i64.sh
-rw-r--r-- 1 spack_cpu spack  26K Jul 10 16:41 spark@3.4.0.o22902088.exp-15-02
-rw-r--r-- 1 spack_cpu spack 2.2K Jul 10 16:41 py-htseq@0.11.2.sh
-rw-r--r-- 1 spack_cpu spack  25K Jul 10 16:46 spark@3.4.0.o23942033.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.2K Jul 10 17:17 spark@3.4.0.sh
drwxr-sr-x 7 spack_cpu spack  268 Jul 10 17:19 .
-rw-r--r-- 1 spack_cpu spack  24K Jul 10 17:21 spark@3.4.0.o23942559.exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$ tail -n 10 spark@3.4.0.o23942559.exp-15-56
==> No patches needed for spark
==> spark: Executing phase: 'install'
==> spark: Successfully installed spark-3.4.0-sntnkcqkor2zmovkogcjf5qivzxdul5v
  Fetch: 42.42s.  Build: 2.84s.  Total: 45.26s.
[+] /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/spark-3.4.0-sntnkcqkor2zmovkogcjf5qivzxdul5v
real 55.59
user 11.73
sys 2.01
==> Regenerating lmod module files
==> OpenFOAM bashrc env: /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/aocc-3.2.0/openfoam-2106-jz42us227mirxrhqjvojlaut2giuh74j/etc/bashrc
[spack_cpu@exp-15-56 gcc@10.2.0]$
mkandes commented 1 year ago
[mkandes@login02 ~]$ spack find -lvdp spark
==> 2 installed packages
-- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
po6mvtn spark@3.4.0+hadoop  /home/mkandes/.spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/spark-3.4.0-po6mvtnx2dirat72biironfv4ikcw5m2
bfyu354     hadoop@3.3.0               /home/mkandes/.spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/hadoop-3.3.0-bfyu354o4z5trshyxr6zkg463zdoat2f
27cv2ps         openjdk@11.0.12_7      /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/openjdk-11.0.12_7-27cv2psb6xthjbwmjlx53otdbjxch6wg

sntnkcq spark@3.4.0+hadoop  /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/spark-3.4.0-sntnkcqkor2zmovkogcjf5qivzxdul5v
bfyu354     hadoop@3.3.0               /home/mkandes/.spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/hadoop-3.3.0-bfyu354o4z5trshyxr6zkg463zdoat2f
27cv2ps         openjdk@11.0.12_7      /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/openjdk-11.0.12_7-27cv2psb6xthjbwmjlx53otdbjxch6wg

[mkandes@login02 ~]$
mkandes commented 1 year ago

Note, however, the module is not immediately visible for some reason ...

[mkandes_test@login02]~% module spider spark

----------------------------------------------------------------------------
  spark:
----------------------------------------------------------------------------
     Versions:
        spark/2.4.8
        spark/3.2.1

----------------------------------------------------------------------------
  For detailed information about a specific "spark" package (including how to load the modules) use the module's full name. Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider spark/3.2.1
----------------------------------------------------------------------------

[mkandes_test@login02]~%
[mkandes@login02 ~]$ ls -lahtr /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/gcc/10.2.0/spark/3.4.0/
total 2.5K
drwxr-sr-x 3 spack_cpu spack    1 Jul 10 17:21 ..
drwxr-sr-x 2 spack_cpu spack    1 Jul 10 17:21 .
-rw-r--r-- 1 spack_cpu spack 2.4K Jul 10 17:32 sntnkcq.lua
[mkandes@login02 ~]$
mkandes commented 1 year ago

It was discovered that both login nodes currently have a system spider cache that is preventing an updated view of the available modules ...

[mkandes@login02 ~]$ ls -lahtr /var/lib/lmod/mData/cacheDir
total 6.6M
-rw-r--r-- 1 root root 2.0M Jun 19 14:02 spiderT.old.lua
-rw-r--r-- 1 root root 1.4M Jun 19 14:02 spiderT.old.luac_5.3
-rw-r--r-- 1 root root 2.0M Jun 19 14:02 spiderT.lua
-rw-r--r-- 1 root root 1.4M Jun 19 14:02 spiderT.luac_5.3
drwxr-xr-x 2 root root 4.0K Jun 19 14:02 .
drwxr-xr-x 3 root root 4.0K Jun 19 14:02 ..
[mkandes@login02 ~]$

However, this can be ignored with the --ignore_cache flag.

[spack_cpu@login02 ~]$ module --ignore_cache spider spark

----------------------------------------------------------------------------------------------------------------------------------------------------
  spark:
----------------------------------------------------------------------------------------------------------------------------------------------------
     Versions:
        spark/2.4.8
        spark/3.2.1
     Other possible modules matches:
        spark/3.4.0

----------------------------------------------------------------------------------------------------------------------------------------------------
  To find other possible module matches execute:

      $ module -r spider '.*spark.*'

----------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "spark" package (including how to load the modules) use the module's full name. Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider spark/3.2.1
----------------------------------------------------------------------------------------------------------------------------------------------------

[spack_cpu@login02 ~]$
mkandes commented 1 year ago

To finalize this production deployment of Spark, we'll commit final changes to spec build script back to sdsc/repo, remove test build output and replace with final build output.

[spack_cpu@exp-15-56 gcc@10.2.0]$ git diff
diff --git a/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
index 5234329413..d70c1f03ae 100644
--- a/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
+++ b/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.sh
@@ -2,7 +2,7 @@

 #SBATCH --job-name=spark@3.4.0
 #SBATCH --account=use300
-##SBATCH --reservation=rocky8u7_testing
+#SBATCH --reservation=root_73
 #SBATCH --partition=ind-shared
 #SBATCH --nodes=1
 #SBATCH --ntasks-per-node=1
[spack_cpu@exp-15-56 gcc@10.2.0]$
[spack_cpu@exp-15-56 gcc@10.2.0]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   spark@3.4.0.sh

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    ../../../../../../../compilers.yaml
    ../../../../../../../licenses/aocc/
    ../../../../../../../licenses/intel/
    ../../../../../../../modules.yaml
    ../../../../../../../packages.yaml
    spark@3.4.0.o23942033.exp-15-56
    spark@3.4.0.o23942559.exp-15-56

no changes added to commit (use "git add" and/or "git commit -a")
[spack_cpu@exp-15-56 gcc@10.2.0]$
[spack_cpu@exp-15-56 gcc@10.2.0]$ ls -lahtr | tail -n 10
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 08:53 openblas@0.3.18-omp.sh
-rw-r--r-- 1 spack_cpu spack 2.2K Apr 21 08:54 openblas@0.3.18.sh
-rw-r--r-- 1 spack_cpu spack  27K Apr 21 08:56 openblas@0.3.18-i64.o21819638.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.1K Apr 21 09:00 openblas@0.3.18-i64.sh
-rw-r--r-- 1 spack_cpu spack  26K Jul 10 16:41 spark@3.4.0.o22902088.exp-15-02
-rw-r--r-- 1 spack_cpu spack 2.2K Jul 10 16:41 py-htseq@0.11.2.sh
-rw-r--r-- 1 spack_cpu spack  25K Jul 10 16:46 spark@3.4.0.o23942033.exp-15-56
-rw-r--r-- 1 spack_cpu spack 2.2K Jul 10 17:17 spark@3.4.0.sh
drwxr-sr-x 7 spack_cpu spack  268 Jul 10 17:19 .
-rw-r--r-- 1 spack_cpu spack  24K Jul 10 17:21 spark@3.4.0.o23942559.exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$
[spack_cpu@exp-15-56 gcc@10.2.0]$ git rm spark@3.4.0.o22902088.exp-15-02
rm 'etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/spark@3.4.0.o22902088.exp-15-02'
[spack_cpu@exp-15-56 gcc@10.2.0]$ rm spark@3.4.0.o23942033.exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$ git add spark@3.4.0.sh
[spack_cpu@exp-15-56 gcc@10.2.0]$ git add spark@3.4.0.o23942559.exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$ git status
On branch sdsc-0.17.3
Your branch is up to date with 'origin/sdsc-0.17.3'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    renamed:    spark@3.4.0.o22902088.exp-15-02 -> spark@3.4.0.o23942559.exp-15-56
    modified:   spark@3.4.0.sh

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    ../../../../../../../compilers.yaml
    ../../../../../../../licenses/aocc/
    ../../../../../../../licenses/intel/
    ../../../../../../../modules.yaml
    ../../../../../../../packages.yaml

[spack_cpu@exp-15-56 gcc@10.2.0]$
[spack_cpu@exp-15-56 gcc@10.2.0]$ git commit
[sdsc-0.17.3 bd28762c3a] Deploy spark@3.4.0 % gcc@10.2.0 into prod within expanse/0.17.3/cpu/b
 2 files changed, 93 insertions(+), 100 deletions(-)
 rename etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/{spark@3.4.0.o22902088.exp-15-02 => spark@3.4.0.o23942559.exp-15-56} (72%)
[spack_cpu@exp-15-56 gcc@10.2.0]$
[spack_cpu@exp-15-56 gcc@10.2.0]$ git log
commit bd28762c3a74ea7b511b6c4abbedb3ab50b6ec48 (HEAD -> sdsc-0.17.3)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Tue Jul 11 16:56:09 2023 -0700

    Deploy spark@3.4.0 % gcc@10.2.0 into prod within expanse/0.17.3/cpu/b

commit 553657e0d4e92fb543587583665818cef36d4b94 (origin/sdsc-0.17.3, origin/HEAD)
Merge: e9bb2189ca 7e3d045306
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jul 10 17:15:28 2023 -0700

    Merge pull request #90 from mkandes/sdsc-0.17.3

    Add custom spark package to sdsc package repo in sdsc-0.17.3

commit 7e3d0453067e6d31c67922c47683148a9c5192e6
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Mon Jul 10 17:11:31 2023 -0700

    Add custom spark package to sdsc package repo in sdsc-0.17.3

    The custom change here is simply to include the latest version of
[spack_cpu@exp-15-56 gcc@10.2.0]$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com': 
Enumerating objects: 24, done.
Counting objects: 100% (24/24), done.
Delta compression using up to 128 threads
Compressing objects: 100% (9/9), done.
Writing objects: 100% (13/13), 8.50 KiB | 8.50 MiB/s, done.
Total 13 (delta 5), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (5/5), completed with 5 local objects.
To https://github.com/sdsc/spack.git
   553657e0d4..bd28762c3a  sdsc-0.17.3 -> sdsc-0.17.3
[spack_cpu@exp-15-56 gcc@10.2.0]$
mkandes commented 1 year ago

Deployment complete. https://github.com/sdsc/spack/commit/bd28762c3a74ea7b511b6c4abbedb3ab50b6ec48

In general, what should follow next, if time permits and tests exist, is testing the newly deployed package in production before closing out the issue.