search

sdsc / spack

A flexible package manager that supports multiple versions, configurations, platforms, and compilers.
https://spack.io
Other
1 stars 5 forks source link

ZEN-38311: PKG/SPEC - expanse/0.17.3/cpu/b - Update STAR to v2.7.11b #129

Open mkandes opened 1 month ago

mkandes commented 1 month ago

User requested [1] if we could update STAR [2] on Expanse. An initial look at STAR last week indicated that it should be possible to deploy if via Spack v0.17.3 by simply updating the package.py file to include the new version [3] since there are almost no dependencies or variants to consider.

[1] https://sdsc.zendesk.com/agent/tickets/38311 [2] https://github.com/alexdobin/STAR [3]

[spack_cpu@login02 ~]$ spack info star
Package:   star

Description:
    STAR is an ultrafast universal RNA-seq aligner.

Homepage: https://github.com/alexdobin/STAR

Externally Detectable: 
    False

Tags: 
    None

Preferred version:  
    2.7.10a    https://github.com/alexdobin/STAR/archive/2.7.10a.tar.gz

Safe versions:  
    2.7.10a    https://github.com/alexdobin/STAR/archive/2.7.10a.tar.gz
    2.7.9a     https://github.com/alexdobin/STAR/archive/2.7.9a.tar.gz
    2.7.6a     https://github.com/alexdobin/STAR/archive/2.7.6a.tar.gz
    2.7.0e     https://github.com/alexdobin/STAR/archive/2.7.0e.tar.gz
    2.7.0d     https://github.com/alexdobin/STAR/archive/2.7.0d.tar.gz
    2.6.1b     https://github.com/alexdobin/STAR/archive/2.6.1b.tar.gz
    2.6.1a     https://github.com/alexdobin/STAR/archive/2.6.1a.tar.gz
    2.6.0c     https://github.com/alexdobin/STAR/archive/2.6.0c.tar.gz
    2.6.0b     https://github.com/alexdobin/STAR/archive/2.6.0b.tar.gz
    2.6.0a     https://github.com/alexdobin/STAR/archive/2.6.0a.tar.gz
    2.5.4b     https://github.com/alexdobin/STAR/archive/2.5.4b.tar.gz
    2.5.4a     https://github.com/alexdobin/STAR/archive/2.5.4a.tar.gz
    2.5.3a     https://github.com/alexdobin/STAR/archive/2.5.3a.tar.gz
    2.5.2b     https://github.com/alexdobin/STAR/archive/2.5.2b.tar.gz
    2.5.2a     https://github.com/alexdobin/STAR/archive/2.5.2a.tar.gz
    2.4.2a     https://github.com/alexdobin/STAR/archive/STAR_2.4.2a.tar.gz

Deprecated versions:  
    None

Variants:
    None

Installation Phases:
    install

Build Dependencies:
    zlib

Link Dependencies:
    zlib

Run Dependencies:
    None

Virtual Packages: 
    None

[spack_cpu@login02 ~]$
mkandes commented 1 month ago

Clone fork and create new feature branch.

mkandes@hardtack:~$ git clone https://github.com/mkandes/spack.git
Cloning into 'spack'...
remote: Enumerating objects: 568174, done.
remote: Counting objects: 100% (483/483), done.
remote: Compressing objects: 100% (212/212), done.
remote: Total 568174 (delta 191), reused 483 (delta 191), pack-reused 567691 (from 1)
Receiving objects: 100% (568174/568174), 246.30 MiB | 47.50 MiB/s, done.
Resolving deltas: 100% (259511/259511), done.
Updating files: 100% (12439/12439), done.
mkandes@hardtack:~$ cd spack/
mkandes@hardtack:~/spack$ git branch
* sdsc-0.17.3
mkandes@hardtack:~/spack$ git log -1
commit 1189ec74627ea872a5b5ed918cff7cf1368722a1 (HEAD -> sdsc-0.17.3, origin/sdsc-0.17.3, origin/HEAD)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Tue Aug 27 16:22:24 2024 -0700

    Deploy intel-tbb@2020.3 % intel@19.1.3.304 to exp/0.17.3/cpu/b
mkandes@hardtack:~/spack$ git branch -a
* sdsc-0.17.3
  remotes/origin/HEAD -> origin/sdsc-0.17.3
  remotes/origin/develop
  remotes/origin/sdsc-0.17.3
  remotes/origin/sdsc-0.17.3-gh-124-inst-exp-cpu-dev
  remotes/origin/sdsc-0.17.3-gh-124-inst-exp-gpu-dev
  remotes/origin/sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf
  remotes/origin/sdsc-0.17.3-gh-127-spec-exp-cpu-intel-tbb
mkandes@hardtack:~/spack$ git checkout -b sdsc-0.17.3-gh-129-pkg-spec-exp-cpu-star
Switched to a new branch 'sdsc-0.17.3-gh-129-pkg-spec-exp-cpu-star'
mkandes@hardtack:~/spack$ git branch
  sdsc-0.17.3
* sdsc-0.17.3-gh-129-pkg-spec-exp-cpu-star
mkandes@hardtack:~/spack$
mkandes commented 1 month ago

STAR is already a custom package.py file in the SDSC package repo. I do not recall what the custom changes were here.

mkandes@hardtack:~/spack$ cd var/spack/repos/
mkandes@hardtack:~/spack/var/spack/repos$ ls
builtin  builtin.mock  sdsc  tscc  tutorial
mkandes@hardtack:~/spack/var/spack/repos$ cd sdsc/
mkandes@hardtack:~/spack/var/spack/repos/sdsc$ ls
packages  repo.yaml
mkandes@hardtack:~/spack/var/spack/repos/sdsc$ cd packages/
mkandes@hardtack:~/spack/var/spack/repos/sdsc/packages$ ls
amber         aocc         intel      openmpi           relion   vmd
amdblis       aocl-sparse  libbeagle  orca              spark    wannier90
amdfftw       gamess       matlab     perl              star
amdlibflame   gaussian     mbedtls    py-pyscf          tcl
amdlibm       gdbm         mvapich2   qchem             trinity
amdscalapack  gromacs      nwchem     quantum-espresso  vasp6
mkandes@hardtack:~/spack/var/spack/repos/sdsc/packages$ cd star/
mkandes@hardtack:~/spack/var/spack/repos/sdsc/packages/star$ ls
package.py
mkandes@hardtack:~/spack/var/spack/repos/sdsc/packages/star$
mkandes commented 1 month ago

It looks like the only change was similar to this one. Include new version.

mkandes@hardtack:~/spack/var/spack/repos/sdsc/packages/star$ diff package.py ../../../builtin/packages/star/package.py 
15,16d14
<     version('2.7.10a', sha256='af0df8fdc0e7a539b3ec6665dce9ac55c33598dfbc74d24df9dae7a309b0426a')
<     version('2.7.9a', sha256='ff52c9d6daaa9fb7261efa3aa49ef6ce5262aa089b0762a3cbc751e81321050e')
mkandes@hardtack:~/spack/var/spack/repos/sdsc/packages/star$
mkandes commented 1 month ago

Added latest versions of STAR to updated package.py file.

mkandes@hardtack:~/spack/var/spack/repos/sdsc/packages/star$ git diff package.py
diff --git a/var/spack/repos/sdsc/packages/star/package.py b/var/spack/repos/sdsc/packages/star/package.py
index ccd4268ab7..3f83fd1f49 100644
--- a/var/spack/repos/sdsc/packages/star/package.py
+++ b/var/spack/repos/sdsc/packages/star/package.py
@@ -12,6 +12,10 @@ class Star(Package):
     homepage = "https://github.com/alexdobin/STAR"
     url      = "https://github.com/alexdobin/STAR/archive/2.7.6a.tar.gz"

+
+    version('2.7.11b', sha256='3f65305e4112bd154c7e22b333dcdaafc681f4a895048fa30fa7ae56cac408e7')
+    version('2.7.11a', sha256='542457b1a4fee73f27a581b1776e9f73ad2b4d7e790388b6dc71147bd039f99a')
+    version('2.7.10b', sha256='0d1b71de6c5be1c5d90b32130d2abcd5785a4fc7c1e9bf19cc391947f2dc46e5')
     version('2.7.10a', sha256='af0df8fdc0e7a539b3ec6665dce9ac55c33598dfbc74d24df9dae7a309b0426a')
     version('2.7.9a', sha256='ff52c9d6daaa9fb7261efa3aa49ef6ce5262aa089b0762a3cbc751e81321050e')
     version('2.7.6a', sha256='9320797c604673debea0fe8f2e3762db364915cc59755de1a0d87c8018f97d51')
mkandes@hardtack:~/spack/var/spack/repos/sdsc/packages/star$
mkandes commented 1 month ago

Added star@2.7.11b % gcc@10.2.0 spec build script to exp/0.17.3/cpu/b

mkandes@hardtack:~/spack/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0$ cp star@2.7.10a.sh star@2.7.11b.sh 
mkandes@hardtack:~/spack/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0$ vi star@2.7.11b.sh 
mkandes@hardtack:~/spack/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0$ git add star@2.7.11b.sh 
mkandes@hardtack:~/spack/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0$ git commit
[sdsc-0.17.3-gh-129-pkg-spec-exp-cpu-star 799e7ded9b] Add star@2.7.11b % gcc@10.2.0 to exp/0.17.3/cpu/b
 1 file changed, 67 insertions(+)
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/star@2.7.11b.sh
mkandes@hardtack:~/spack/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0$ git push
Username for 'https://github.com': mkandes
Password for 'https://mkandes@github.com': 
Enumerating objects: 22, done.
Counting objects: 100% (22/22), done.
Delta compression using up to 8 threads
Compressing objects: 100% (9/9), done.
Writing objects: 100% (12/12), 1.62 KiB | 830.00 KiB/s, done.
Total 12 (delta 4), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To https://github.com/mkandes/spack.git
   4effd11498..799e7ded9b  sdsc-0.17.3-gh-129-pkg-spec-exp-cpu-star -> sdsc-0.17.3-gh-129-pkg-spec-exp-cpu-star
mkandes@hardtack:~/spack/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0$ git log -2
commit 799e7ded9b2a13a52e708f8010d75deee4f1d4a6 (HEAD -> sdsc-0.17.3-gh-129-pkg-spec-exp-cpu-star, origin/sdsc-0.17.3-gh-129-pkg-spec-exp-cpu-star)
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Fri Sep 20 15:51:47 2024 -0700

    Add star@2.7.11b % gcc@10.2.0 to exp/0.17.3/cpu/b

commit 4effd114980758dd367345ff42d433bb94b3d122
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Fri Sep 20 15:44:57 2024 -0700

    Update package.py for STAR within custom sdsc package repo

    See https://github.com/sdsc/spack/issues/129
mkandes@hardtack:~/spack/etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0$
mkandes commented 1 month ago

Pull request created and merged. https://github.com/sdsc/spack/pull/130

mkandes commented 1 month ago

Pull changes to production instance.

[spack_cpu@exp-15-56 b]$ git pull
remote: Enumerating objects: 22, done.
remote: Counting objects: 100% (22/22), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 22 (delta 9), reused 21 (delta 9), pack-reused 0 (from 0)
Unpacking objects: 100% (22/22), 3.34 KiB | 103.00 KiB/s, done.
From https://github.com/sdsc/spack
   1189ec7462..6065106649  sdsc-0.17.3 -> origin/sdsc-0.17.3
Updating 1189ec7462..6065106649
Fast-forward
 .../0.17.3/cpu/b/specs/gcc@10.2.0/star@2.7.11b.sh  | 67 ++++++++++++++++++++++
 var/spack/repos/sdsc/packages/star/package.py      |  4 ++
 2 files changed, 71 insertions(+)
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/gcc@10.2.0/star@2.7.11b.sh
[spack_cpu@exp-15-56 b]$ git log -2
commit 6065106649234486cad73e7ef30fe095de5b414d (HEAD -> sdsc-0.17.3, origin/sdsc-0.17.3, origin/HEAD)
Merge: 1189ec7462 799e7ded9b
Author: Marty Kandes, Ph.D. <mkandes@sdsc.edu>
Date:   Fri Sep 20 15:57:57 2024 -0700

    Merge pull request #130 from mkandes/sdsc-0.17.3-gh-129-pkg-spec-exp-cpu-star

    Add star@2.7.11b % gcc@10.2.0 to expanse/0.17.3/cpu/b

commit 799e7ded9b2a13a52e708f8010d75deee4f1d4a6
Author: Marty Kandes <mkandes@sdsc.edu>
Date:   Fri Sep 20 15:51:47 2024 -0700

    Add star@2.7.11b % gcc@10.2.0 to exp/0.17.3/cpu/b
[spack_cpu@exp-15-56 b]$
mkandes commented 1 month ago

Deploying package spec.

[spack_cpu@exp-15-56 gcc@10.2.0]$ sbatch star@2.7.11b.sh 
Submitted batch job 34015247
[spack_cpu@exp-15-56 gcc@10.2.0]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          34015247 ind-share star@2.7 spack_cp  R       0:09      1 exp-15-56
          34015042 ind-share     bash spack_cp  R      10:09      1 exp-15-56
[spack_cpu@exp-15-56 gcc@10.2.0]$
mkandes commented 1 month ago

Build looks successful. 

==> Fetching https://mirror.spack.io/_source-cache/archive/3f/3f65305e4112bd154c7e22b333dcdaafc681f4a895048fa30fa7ae56cac408e7.tar.gz
==> No patches needed for star
==> star: Executing phase: 'install'
==> star: Successfully installed star-2.7.11b-c27utuxywta6qft47zzgghmxj3wnt7ba
  Fetch: 1.00s.  Build: 51.23s.  Total: 52.22s.
[+] /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/star-2.7.11b-c27utuxywta6qft47zzgghmxj3wnt7ba
real 54.73
user 170.49
sys 254.07
[spack_cpu@exp-15-56 gcc@10.2.0]$ spack find -lvd star@2.7.11b
==> 1 installed package
-- linux-rocky8-zen2 / gcc@10.2.0 -------------------------------
c27utux star@2.7.11b
ws4iari     zlib@1.2.11+optimize+pic+shared

[spack_cpu@exp-15-56 gcc@10.2.0]$
mkandes commented 1 month ago

Deployment complete after module refresh. User notified for testing.

[mkandes@login02 ~]$ module load gcc/10.2.0
[mkandes@login02 ~]$ module load star/2.7.11b
[mkandes@login02 ~]$ which start
start-statd  startx       
[mkandes@login02 ~]$ which STAR
/cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/star-2.7.11b-c27utuxywta6qft47zzgghmxj3wnt7ba/bin/STAR
[mkandes@login02 ~]$
mkandes commented 1 month ago

https://github.com/sdsc/spack/commit/b62c4b176889a485e7ae0c2e857c7613e302afa6