CASTEP 23.1.1 and CASTEP 24.1

[heatherkellyucl]

The existing recipe covers CASTEP 19.1.1.rc2 and 21.11.

I have added the checksum for 23.1.1 in repos/dev/packages/castep/package.py

Need to check if there are any flags or variants needed for this version, if it compiles and if it works.

[heatherkellyucl]

I've added these variants:

    variant("grimmed3", default=False, description="Enable Grimme DFT+D library")

    variant("grimmed4", default=False, when="@23:", description="Enable Grimme D4 library")

    variant(
        "libxc", default=False, when="@23:",
        description="Enable libxc library of additional XC functionals"
    )

and some depends_on, around the existing mpi one:

    depends_on("perl", type=("build", "run"))
    depends_on("mpi", type=("build", "link", "run"), when="+mpi")
    depends_on("libxc", type=("build", "link", "run"), when="+libxc")

(not sure if libxc is needed at runtime) and this is where it adds them to the build options:

        # "system" for existing libxc, "compile" to build own
        if "+libxc" in spec:
            targetlist.append("LIBXC=system")

        if "+grimmed3" in spec:
            targetlist.append("GRIMMED3=compile")

        if "+grimmed4" in spec:
            targetlist.append("GRIMMED4=compile")

Can't enable the xml variant right now because it requires FoX: A Fortran library for XML processing which isn't in Spack right now. The only fox is a C++ toolkit. (Quantum Espresso builds its own FoX). So it would need a different name if I added it.

CASTEP has a cmake build system too - probably that one should be added as well (https://spack.readthedocs.io/en/latest/packaging_guide.html#writing-a-package-recipe) but not right now.

I don't think the recipe builds castep.serial right now, which we want it to do. You'd normally do a serial build as well as the mpi build and install them both into the same place. To check.

[heatherkellyucl]

Altered the new variants above to add the when="@23:" since they were added in that version.

I was hoping if our source mirror was there then it would look in it for the tar file automatically - that doesn't appear to be the case so another bit of config needs adjusting. For right now, the package recipe tells it to look in the current directory for the source, so copying in there works.

However, part of the filter is wrong for this version:

==> Installing castep-23.1.1-2h2cflxs6gt2prrzr7kzuod7tktms4d5 [96/100]
==> No binary for castep-23.1.1-2h2cflxs6gt2prrzr7kzuod7tktms4d5 found: installing from source
==> Fetching file:///lustre/scratch/scratch/ccspapp/spack/0.22/hpc-spack/castep-23.1.1.tar.bz2
==> No patches needed for castep
==> castep: Executing phase: 'edit'
==> Error: FileNotFoundError: [Errno 2] No such file or directory: 'LibSource/dl_mg-2.0.3/platforms/castep.inc
'

/lustre/scratch/scratch/ccspapp/spack/0.22/hpc-spack/repos/dev/packages/castep/package.py:67, in edit:
         64    def edit(self, spec, prefix):
         65        if spec.satisfies("%gcc"):
         66            dlmakefile = FileFilter("LibSource/dl_mg-2.0.3/platforms/castep.inc")
  >>     67            dlmakefile.filter(r"MPIFLAGS = -DMPI", "MPIFLAGS = -fallow-argument-mismatch -DMPI")
         68            if self.spec.satisfies("@20:"):
         69                platfile = FileFilter("obj/platforms/linux_x86_64_gfortran.mk")
         70            else:

See build log for details:
  /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-2h2cflxs6gt2prrzr7kzuod7tktms4d5/spack-build-out.txt

It is because it is LibSource/dl_mg-3.0.0 in this version, need to add logic for that.

[heatherkellyucl]

I confirmed that dl_mg-3.0.0 does also need -fallow-argument-mismatch adding to the MPIFLAGS.

Changed to:

            if self.spec.satisfies("@19:21"):
                dlmakefile = FileFilter("LibSource/dl_mg-2.0.3/platforms/castep.inc")
            if self.spec.satisfies("@23:"):
                dlmakefile = FileFilter("LibSource/dl_mg-3.0.0/platforms/castep.inc")
            dlmakefile.filter(r"MPIFLAGS = -DMPI", "MPIFLAGS = -fallow-argument-mismatch -DMPI")

[heatherkellyucl]

Still got type mismatches.

==> Error: ProcessError: Command exited with status 2:
    'make' '-j6' 'PWD=/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-cast
ep-23.1.1-2h2cflxs6gt2prrzr7kzuod7tktms4d5/spack-src' 'COMMS_ARCH=mpi' 'FFTLIBDIR=/lustre/shared/ucl/apps/spac
k/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_pla
ceholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_place
holder__/linux-rhel7-cascadelake/gcc-12.3.0/fftw-3.3.10-3hrfzp6xsbvyfdebwta53swqul7tt3x4/lib' 'MATHLIBDIR=/lus
tre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeh
older__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placehol
der__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/openblas-0.3.26-mtr3n5zbgfqoca4pjp5xx5gotc5dqvhy/lib' 'FFT=fftw3' 'MATHLIBS=openblas' 'ARCH=linux_x86_64_gfortran'

242 errors found in build log:
     262
     263     22418 |        call MPI_allreduce(local_array,z_array(bufpos),min(buflen,length-bufpos+1),&
     264           |                          1
     265     ......
     266     22517 |        call MPI_allreduce(local_array,d_array(bufpos),min(buflen,length-bufpos+1),&
     267           |                          2
  >> 268     Error: Type mismatch between actual argument at (1) and actual argument at (2) (COMPLEX(8)/REAL(
             8)).
     269     ../../../Source/Utility/comms.mpi.F90:22418:38:
...

  >> 2180    Error: Type mismatch between actual argument at (1) and actual argument at (2) (CHARACTER(12)/IN
             TEGER(4)).
  >> 2181    make[1]: *** [../../include_rules.mk:155: comms.mpi.anc] Error 1
...

See build log for details:
  /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-2h2cflxs6gt2prrzr7kzuod7tktms4d5/spack-build-out.txt

[giordano]

You can try adding fflags=-fallow-argument-mismatch to the spec as an alternative, which forces the flag in the wrapper of the fortran compiler used by spack.

[heatherkellyucl]

I'll have a go like that temporarily.

It looks like it needs adding in the package.py to FFLAGS_FOR_LIBRARIES in obj/platforms/linux_x86_64_gfortran.mk since it already gets added to FFLAGS_E which are the general flags.

# Flags for building third-party libraries (DL_MG, D3, D4). Assign with "=" to include $(OPT) defined later
#
FFLAGS_FOR_LIBRARIES = -fconvert=big-endian -fopenmp -fPIC -ffpe-summary=none $(OPT)

[heatherkellyucl]

Though there is a linux_x86_64_gfortran10.mk so maybe we should be using that now.

[heatherkellyucl]

Including in spec worked, as castep@23.1.1 fflags=-fallow-argument-mismatch:

==> Installing castep-23.1.1-odprczkj44tekqlbv4rwbzpdfecfsvel [96/101]
==> No binary for castep-23.1.1-odprczkj44tekqlbv4rwbzpdfecfsvel found: installing from source
==> Using cached archive: /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/cache/_source-ca
che/archive/8d/8d922c641c99fc6e4f5b4f7f2478abab897065850e454fb4968154ddbb566388.tar.bz2
==> No patches needed for castep
==> castep: Executing phase: 'edit'
==> castep: Executing phase: 'build'
==> castep: Executing phase: 'install'
==> castep: Successfully installed castep-23.1.1-odprczkj44tekqlbv4rwbzpdfecfsvel
  Stage: 26.42s.  Edit: 0.20s.  Build: 4m 20.51s.  Install: 1m 2.18s.  Post-install: 3.08s.  Total: 6m 3.66s

[heatherkellyucl]

There's a linux_x86_64_gfortran10.mk in our 20.1 source as well, so it looks like it has existed since then and should probably be the one used in all versions after 19.x. Assuming that works.

[heatherkellyucl]

Working out how to include this lot of recommended and incompatible compilers is going to be interesting...

 USE: The following compilers have been tested and confirmed to
 correctly compile CASTEP 23.1.1

 *  Intel Fortran 18.0.2+, 19.0.0, 19.0.2
 *  Intel oneAPI Fortran (classic) 2021.2.0-2021.3.0, 2021.5.0-2021.6.0
    (packaged as oneAPI version 2021.2,2021.3,2022.0.1,2022.0.2,2022.1.0)
 *  GNU Fortran   7.2-7.5, 8.*, 9.2, 9.3.0, 10.1.0, 11.x, 12.x
 *  NVIDIA/PGI    nvhpc 20.7-21.X, 22.5, 22.11 (+)
 *  NAG Fortran   7.1
 *  Cray Fortran  11.0.4, 12.0.3, 13.0.2 (++)

 AVOID: The following compilers either have bugs which result in at least one
 failure, either during compilation or a crash or incorrect
 results at runtime, or do not support the language standards required.
 There may be more details of the seriousness of
 the failures in the platform-specific sections below.

 *  Intel Fortran < 18.0, 18.0.0, 18.0.1 (*), 19.1.0(**)
 *  Intel oneAPI - ifx (any version) The new flang-based compiler (ifx) 
    suffers an ICE on attempted compile.
 *  Intel oneAPI classic (ifort) 2021.4.0, 2021.7.0,2021.7.1
    packaged as oneAPI 2021.4, 2022.2.0, 2022.2.1
    Various run-time failures on the test suite.
 *  NVidia nvfortran 22.7-22.10
 *  AMD AOCC flang < 4.0 (***)
 *  PGI Fortran < 19.x
 *  Oracle/Sun fortran (all versions) (****)
 *  Cray Fortran  11.0.x
 *  AMD AOCC flang (++)

For footnotes see README.INSTALL.

[heatherkellyucl]

Looking at https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/quantum-espresso/package.py for examples because it has lots of different kinds of conflicts.

    # Internal compiler error gcc8 and a64fx, I check only 6.5 and 6.6
    conflicts(
        "@5.3:", when="%gcc@8 target=a64fx", msg="Internal compiler error with gcc8 and a64fx"
    )

    with when("%nvhpc"):
        variant("cuda", default=False, description="Build with CUDA Fortran")
        with when("+cuda"):
            conflicts(
                "%nvhpc@21.11:22.3",
                msg="bugs with NVHPCSDK from v21.11 to v22.3, OpenMP and GPU",
            )

So like this? (https://spack.readthedocs.io/en/latest/basic_usage.html#version-specifier)

with when ("%intel-oneapi-compilers-classic"):
    conflicts(
        "%intel-oneapi-compilers-classic@2021.4.0,2021.7",
        msg="Various run-time failures on the test suite with ifort 2021.4.0, 2021.7.0, 2021.7.1",
    )

[heatherkellyucl]

The build stage with the source and tests are in /shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-2h2cflxs6gt2prrzr7kzuod7tktms4d5/spack-src/Test/ still at the moment.

To run test suite at its simplest, just run

make

in this directory or

make check

in the top-level directory. Tests will be run using the CASTEP executable "obj/$ARCH/castep", ie the default compiled executable. "make check ARCH=xxx" works as expected, running the alternative executable.

The module for this CASTEP install is at /shared/ucl/apps/spack/0.22/hk-initial-stack/modules/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/myriad/linux-rhel7-cascadelake/castep/23.1.1-openmpi/gcc-12.3.0

(Need to fix that somewhere).

[heatherkellyucl]

module use /shared/ucl/apps/spack/0.22/hk-initial-stack/modules/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/myriad/linux-rhel7-cascadelake

module avail castep
- /shared/ucl/apps/spack/0.22/hk-initial-stack/modules/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/myriad/linux-rhel7-cascadelake -
castep/23.1.1-openmpi/gcc-12.3.0  

--------------------------------- /shared/ucl/apps/modulefiles/applications ----------------------------------
castep/17.2/intel-2017  castep/17.21/intel-2017  castep/19.1.1/intel-2019 

[heatherkellyucl]

Needed to make sure gcc@12.3.0 is explicitly added to the environment too, since only the gcc-runtime was there as a module.

Then regenerated modules.

spack -e myriad add gcc@12.3.0
spack -e myriad concretize
spack -e myriad install
spack -e myriad module tcl refresh

module avail gcc
- /shared/ucl/apps/spack/0.22/hk-initial-stack/modules/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/myriad/linux-rhel7-cascadelake -
gcc-runtime/11.2.1/gcc-11.2.1  gcc-runtime/12.3.0/gcc-12.3.0  gcc/12.3.0/gcc-11.2.1  

----------------------------------- /shared/ucl/apps/modulefiles/libraries -----------------------------------
gcc-libs/4.9.2  gcc-libs/7.3.0  gcc-libs/8.3.0  gcc-libs/9.2.0  gcc-libs/10.2.0  

The gcc-runtimes are like our gcc-libs, and the gcc loads some other modules and sets up CC, CXX, FC, F77 like our compilers/gnu modules.

[heatherkellyucl]

I want to build some of the tests, so need the compiler module, plus our already-built castep:

[ccspapp@build01 castep-castep-65582dd54bf3]$ module load gcc/12.3.0/gcc-11.2.1
Loading gcc/12.3.0/gcc-11.2.1
  Loading requirement: glibc/2.17/gcc-11.2.1 gcc-runtime/11.2.1/gcc-11.2.1 zlib/1.3.1/gcc-11.2.1
    zstd/1.5.5/gcc-11.2.1 binutils/2.42/gcc-11.2.1 gmp/6.2.1/gcc-11.2.1 mpfr/4.2.0/gcc-11.2.1
    mpc/1.3.1/gcc-11.2.1
[ccspapp@build01 castep-castep-65582dd54bf3]$ module load castep/23.1.1-openmpi/gcc-12.3.0
Loading castep/23.1.1-openmpi/gcc-12.3.0
  Loading requirement: gcc-runtime/12.3.0/gcc-12.3.0 libpciaccess/0.17/gcc-12.3.0 libiconv/1.17/gcc-11.2.1
    xz/5.4.6/gcc-11.2.1 libxml2/2.10.3/gcc-11.2.1 ncurses/6.4/gcc-11.2.1 hwloc/2.9.1/gcc-12.3.0
    knem/1.1.4/gcc-12.3.0 openssl/3.3.0/gcc-12.3.0 libevent/2.1.12/gcc-12.3.0 libfabric/1.21.0/gcc-12.3.0
    numactl/2.0.14/gcc-12.3.0 opa-psm2/11.2.230/gcc-12.3.0 bzip2/1.0.8/gcc-11.2.1 pigz/2.8/gcc-11.2.1
    tar/1.30/gcc-11.2.1 gettext/0.22.5/gcc-11.2.1 krb5/1.20.1/gcc-12.3.0 libedit/3.1-20230828/gcc-12.3.0
    libxcrypt/4.4.35/gcc-12.3.0 openssh/9.7p1/gcc-12.3.0 pmix/5.0.1/gcc-12.3.0 ucx/1.16.0/gcc-12.3.0
    xpmem/2.6.5-36/gcc-12.3.0 openmpi/4.1.6/gcc-12.3.0 fftw/3.3.10-openmpi/gcc-12.3.0
    openblas/0.3.26/gcc-12.3.0 berkeley-db/18.1.40/gcc-11.2.1 readline/8.2/gcc-11.2.1 gdbm/1.23/gcc-11.2.1
    perl/5.38.0/gcc-11.2.1

[ccspapp@build01 castep-castep-65582dd54bf3]$ castep.mpi
Usage:
castep <seedname>                : Run files <seedname>.cell [and <seedname>.param]
  "    [-d|--dryrun] <seedname>  : Perform a dryrun calculation on files <seedname>.cell
  "    [-s|--search] <text>      : print list of keywords with <text> match in description
  "    [-v|--version]            : print version information
  "    [-h|--help] <keyword>     : describe specific keyword in <>.cell or <>.param
  "         "      all           : print list of all keywords
  "         "      basic         : print list of basic-level keywords
  "         "      inter         : print list of intermediate-level keywords
  "         "      expert        : print list of expert-level keywords
  "         "      dummy         : print list of dummy keywords

[heatherkellyucl]

Going to use the non-build-stage untarring of castep which is in ccspapp's Scratch.

[ccspapp@build01 castep-castep-65582dd54bf3]$ pwd
/home/ccspapp/Scratch/castep/castep-castep-65582dd54bf3

[ccspapp@build01 castep-castep-65582dd54bf3]$ cd Test
[ccspapp@build01 Test]$ make CASTEPEXE=/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/castep-23.1.1-odprczkj44tekqlbv4rwbzpdfecfsvel/bin/castep.mpi
rm -f */*/*.castep */*/*.dfpt_wvfn */*/*.fd_wvfn */*/*.wvfn.* */*/*.*.err
../bin/testcode.py -q  --total-processors=36 -e /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/castep-23.1.1-odprczkj44tekqlbv4rwbzpdfecfsvel/bin/castep.mpi -c simple -c d3-simple -c d4-simple
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... [486/486]

Uh, I think that worked? 486 tests passed.

[heatherkellyucl]

Can look in Dispersion/HeNe_D3/test.out.150724.inp\=HeNe_D3_PBE-geom.param for one of the test outputs.

Ends with

     Hirshfeld Analysis
     ------------------
Species   Ion     Hirshfeld Charge (e)
======================================
  He       1                 0.00
  Ne       1                -0.00
======================================

Writing analysis data to HeNe_D3_PBE-geom.castep_bin

Writing model to HeNe_D3_PBE-geom.check

 A BibTeX formatted list of references used in this run has been written to 
 HeNe_D3_PBE-geom.bib

Initialisation time =      0.82 s
Calculation time    =      9.19 s
Finalisation time   =      0.06 s
Total time          =     10.07 s
Peak Memory Use     = 4983964 kB

[heatherkellyucl]

That was all with the install that was doing castep@23.1.1 fflags=-fallow-argument-mismatch.

I have pushed some changes to repos/dev/packages/castep/package.py that need testing next (and that's going to need some tidying because there are 'if version' combos all over the place).

[heatherkellyucl]

I found why the pkg-store wasn't working, mirrors need a specific structure and don't just look for the archive in that path. <name>/<name>-<version>.<extension>. https://spack.readthedocs.io/en/latest/mirrors.html#mirrors

So on Myriad, I have added castep and orca subdirectories to pkg-store now that have the correct structure and name for the most recent packages. I've left the original package names at the top level for now since our buildscripts use them as they were - should tidy that up so there aren't duplicates.

[heatherkellyucl]

Can try auditing my package to find problems (not sure exactly what it does/doesn't pick up).

spack audit packages castep
PKG-DIRECTIVES: 0 issues found.
PKG-ATTRIBUTES: 0 issues found.
PKG-PROPERTIES: 0 issues found.

Then uninstall my previous build so I can try again with the updated package recipe.

[ccspapp@build01 hpc-spack]$ spack -e myriad uninstall castep
    -- linux-rhel7-cascadelake / gcc@12.3.0 -------------------------
    odprczk castep@23.1.1

==> 1 packages will be uninstalled. Do you want to proceed? [y/N] y
==> Successfully uninstalled castep@23.1.1%gcc@12.3.0~grimmed3~grimmed4~libxc+mpi build_system=makefile arch=linux-rhel7-cascadelake/odprczk

[giordano]

(not sure exactly what it does/doesn't pick up).

It should mainly do static checks, like consistency of dependencies (in case you depend on non-existing versions of a package), or the use deprecated versions, etc.

[heatherkellyucl]

Deleted cached archive so we start fresh again:

rm /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/cache/_source-cache/archive/8d/8d922c641c99fc6e4f5b4f7f2478abab897065850e454fb4968154ddbb566388.tar.bz2

spack -e myriad install

==> Installing castep-23.1.1-odprczkj44tekqlbv4rwbzpdfecfsvel [99/104]
==> No binary for castep-23.1.1-odprczkj44tekqlbv4rwbzpdfecfsvel found: installing from source
==> Fetching file:///shared/ucl/apps/pkg-store/castep/castep-23.1.1.tar.bz2
==> No patches needed for castep
==> castep: Executing phase: 'edit'
==> castep: Executing phase: 'build'
==> castep: Executing phase: 'install'
==> castep: Successfully installed castep-23.1.1-odprczkj44tekqlbv4rwbzpdfecfsvel
  Stage: 29.49s.  Edit: 0.02s.  Build: 4m 32.94s.  Install: 1m 1.82s.  Post-install: 2.16s.  Total: 6m 13.29s

and done, so looks like using the linux_x86_64_gfortran10.mk works fine.

[heatherkellyucl]

I think platfile.filter(r"^\s*FFLAGS_E\s*=.*", "FFLAGS_E = -fallow-argument-mismatch ") is only needed if you are a) building castep 19 or b) building newer castep using a compiler older than gcc 10. linux_x86_64_gfortran10.mk ought to add it in for all the bits that need it for gcc10 onwards.

It might even only be needed for a) building castep 19 with gcc 10+ but I'm not sure of that and would need to check with older compilers that we don't want to use to make sure. (Maybe later).

[heatherkellyucl]

Changed repos/dev/packages/castep/package.py to reflect the top one. Still builds fine with gcc-12.3.0.

The gcc bit currently looks like this:

        if spec.satisfies("%gcc"):
            if self.spec.satisfies("@19:21"):
                dlmakefile = FileFilter("LibSource/dl_mg-2.0.3/platforms/castep.inc")
            elif self.spec.satisfies("@23:"):
                dlmakefile = FileFilter("LibSource/dl_mg-3.0.0/platforms/castep.inc")

            if self.spec.satisfies("@20:"):
                if spec.satisfies("%gcc@10:"):
                    platfile = FileFilter("obj/platforms/linux_x86_64_gfortran10.mk")
                else:
                    platfile = FileFilter("obj/platforms/linux_x86_64_gfortran.mk")
                    platfile.filter(r"^\s*FFLAGS_E\s*=.*", "FFLAGS_E = -fallow-argument-mismatch ")
            elif self.spec.satisfies("@19"):
                if spec.satisfies("%gcc@9:"):
                    platfile = FileFilter("obj/platforms/linux_x86_64_gfortran9.0.mk")
                else:
                    platfile = FileFilter("obj/platforms/linux_x86_64_gfortran.mk")
                dlmakefile.filter(r"MPIFLAGS = -DMPI", "MPIFLAGS = -fallow-argument-mismatch -DMPI")
                platfile.filter(r"^\s*FFLAGS_E\s*=.*", "FFLAGS_E = -fallow-argument-mismatch ")

            platfile.filter(r"^\s*OPT_CPU\s*=.*", "OPT_CPU = ")

[heatherkellyucl]

Need to test the +libxc, +grimmed3 and +grimmed4 variants. libxc is the one that brings in an additional dependency as the other two build their own.

[heatherkellyucl]

Uninstalled current build again. Removed castep@21.1.1 from /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/myriad/spack.yaml or we end up with both.

[ccspapp@build01 hpc-spack]$ spack -e myriad add castep@23.1.1 +libxc
==> Adding castep@23.1.1+libxc to environment myriad
[ccspapp@build01 hpc-spack]$ spack -e myriad concretize
spack -e myriad install

==> Installing castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu [100/104]
==> No binary for castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu found: installing from source
==> Using cached archive: /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/cache/_source-cache/archive/8d/8d922c641c99fc6e4f5b4f7f2478abab897065850e454fb4968154ddbb566388.tar.bz2
==> No patches needed for castep
==> castep: Executing phase: 'edit'
==> castep: Executing phase: 'build'
==> castep: Executing phase: 'install'
==> castep: Successfully installed castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu
  Stage: 28.69s.  Edit: 0.01s.  Build: 4m 22.24s.  Install: 1m 1.65s.  Post-install: 1.28s.  Total: 5m 59.89s

Different hash, and we have the +libxc version:

spack -e myriad find --variants castep
...
==> Installed packages
-- linux-rhel7-cascadelake / gcc@12.3.0 -------------------------
castep@23.1.1~grimmed3~grimmed4+libxc+mpi build_system=makefile
==> 1 installed package

and the module at /shared/ucl/apps/spack/0.22/hk-initial-stack/modules/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/myriad/linux-rhel7-cascadelake/castep/23.1.1-openmpi/gcc-12.3.0

now includes this when it did not before:

    depends-on libxc/6.2.2/gcc-12.3.0

Need to rerun the tests.

[heatherkellyucl]

Those tests are all ok again - need to see if there's anything specific to libxc.

make CASTEPEXE=/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu/bin/castep.mpi 
rm -f */*/*.castep */*/*.dfpt_wvfn */*/*.fd_wvfn */*/*.wvfn.* */*/*.*.err
../bin/testcode.py -q  --total-processors=36 -e /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu/bin/castep.mpi -c simple -c d3-simple -c d4-simple
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... [486/486]

[heatherkellyucl]

Ok, there is Test/XC/libXC/ and make check-libxc (and I should run some with COMMS_ARCH=mpi to do mpi checking).

[heatherkellyucl]

[ccspapp@build01 Test]$ make check-libxc CASTEPEXE=/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu/bin/castep.mpi
rm -f */*/*.castep */*/*.dfpt_wvfn */*/*.fd_wvfn */*/*.wvfn.* */*/*.*.err
../bin/testcode.py -q  --total-processors=36 -e /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu/bin/castep.mpi -c libXC-simple
.. [2/2]

So the libxc tests are fine.

The MPI tests are not happy, probably need to add more options:

[ccspapp@build01 Test]$ make CASTEPEXE=/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu/bin/castep.mpi COMMS_ARCH=mpi
Found mpirun, using PARALLEL_CMD=mpirun -np tc.nprocs
rm -f */*/*.castep */*/*.dfpt_wvfn */*/*.fd_wvfn */*/*.wvfn.* */*/*.*.err
../bin/testcode.py -q  --processors=4 --total-processors=36 --user-option castep run_cmd_template "../../../bin/run_castep_test.pl tc.program tc.args tc.input tc.output mpirun -np tc.nprocs" -e /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu/bin/castep.mpi -c simple -c d3-simple -c d4-simple -c md-parallel
--------------------------------------------------------------------------
WARNING: Could not generate an xpmem segment id for this process'
address space.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: build01   
  Error code: 2 (No such file or directory)
--------------------------------------------------------------------------
[build01.myriad.ucl.ac.uk:132243] 3 more processes have sent help message help-btl-vader.txt / xpmem-make-failed
[build01.myriad.ucl.ac.uk:132243] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

[heatherkellyucl]

Not sure that actually made them fail as I have a MD-parallel/PIMD/test.out.160724-1.inp=si2_pimd.param and full output for that one (I killed it after a bit since I was getting loads of the above message). Will try running again.

[heatherkellyucl]

See https://github.com/hpc/xpmem/issues/46 : XPMEM expects a kernel module and /dev/xpmem present. Not relevant to the functioning of CASTEP so we just don't want the tests to use that.

[heatherkellyucl]

The Makefile looks at PARALLEL_CMD first if set then in our case does the entry for mpirun:

    # Next try mpirun (default from userconfig)
    else ifneq (,$(shell command -v mpirun 2> /dev/null))
      # Turn off CPU binding for OpenMPI to allow concurrent MPI runs
      export OMPI_MCA_hwloc_base_binding_policy=none
      PARALLEL_CMD := mpirun -np tc.nprocs
      $(info Found mpirun, using PARALLEL_CMD=$(PARALLEL_CMD))

[heatherkellyucl]

Ah, we should remove xpmem from our list of fabrics for openmpi and potentially some of the others. (We were just doing everything except the one that doesn't build). We're not setting any preferred fabrics when running either yet.

openmpi@4.1.6~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~singularity~static+vt+wrapper-rpath build_system=autotools fabrics=cma,knem,ofi,psm2,ucx,xpmem romio-filesystem=none schedulers=sge

[heatherkellyucl]

Temporarily, do this

export OMPI_MCA_btl_vader_single_copy_mechanism=cma

Not sure if we have cma usable but it at least exists.

[ccspapp@build01 Test]$ make CASTEPEXE=/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__s
pack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spa
ck_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0
/castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu/bin/castep.mpi COMMS_ARCH=mpi
Found mpirun, using PARALLEL_CMD=mpirun -np tc.nprocs
rm -f */*/*.castep */*/*.dfpt_wvfn */*/*.fd_wvfn */*/*.wvfn.* */*/*.*.err
../bin/testcode.py -q  --processors=4 --total-processors=36 --user-option castep run_cmd_template "../../../bi
n/run_castep_test.pl tc.program tc.args tc.input tc.output mpirun -np tc.nprocs" -e /lustre/shared/ucl/apps/sp
ack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p
laceholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_pla
ceholder__/linux-rhel7-cascadelake/gcc-12.3.0/castep-23.1.1-niix677nq5we7h63kt3mdabhnu4tlfuu/bin/castep.mpi -c
 simple -c d3-simple -c d4-simple -c md-parallel
..............................................................................................................
..............................................................................................................
..............................................................................................................
..............................................................................................................
............................................... [487/487]

Those ran.

[heatherkellyucl]

Needed to remove previous openmpi from buildcache:

rm -r /shared/ucl/apps/spack/0.22/buildcache/build_cache/linux-rhel7-cascadelake/gcc-12.3.0/openmpi-4.1.6/
rm -r /shared/ucl/apps/spack/0.22/buildcache/build_cache/linux-rhel7-cascadelake-gcc-12.3.0-openmpi-4.1.6-mxk55vj2o7xiscknapf65zfzcjxvctu4.spec.json.sig

spack buildcache update-index /shared/ucl/apps/spack/0.22/buildcache

All gone.

[ccspapp@build01 hpc-spack]$ spack buildcache list --allarch openmpi
==> 0 cached builds.

[heatherkellyucl]

While that is building, am looking at the README.cmake, and hmm, it says that GRIMMED3 and GRIMMED4 are on by default in that. I wonder if they are for the make build as well in this version? Because that will need the recipe adjusting to indicate so.

And the tests above did -c d3-simple -c d4-simple which are indeed the tests for that. Need to check which version that happened in.

[heatherkellyucl]

Buildcache on Myriad updated.

[heatherkellyucl]

Now, why are the modules for this environment being created in

/shared/ucl/apps/spack/0.22/hk-initial-stack/modules/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/myriad/

instead of something like /shared/ucl/apps/spack/0.22/hk-initial-stack/modules/myriad ?

Seems to have $site_root and $spack_root both in there.

[heatherkellyucl]

In https://github.com/UCL-ARC/hpc-spack/blob/0.22/spacksites/settings/initial_site_modules.yaml it has

    roots:
      tcl: $spack/../modules/$env

and that $env is what is giving us the full path rather than $env_name.

[heatherkellyucl]

Was confirmed in Spack Slack as a bug. I'll leave it be for now and if needed we can change what happens with initial_site_modules.yaml, this is where it gets copied into the site: https://github.com/UCL-ARC/hpc-spack/blob/8271867709595d6a11b1f6a3f64dd6caffe7f215/spacksites/src/spacksite.py#L55

If we needed to change it ourselves, it would need to be injected into the environment's spack.yaml I think, so it would govern it per environment.

[heatherkellyucl]

Useful links for multi-build packages:

https://spack.readthedocs.io/en/latest/packaging_guide.html#multiple-build-systems https://spack.readthedocs.io/en/latest/build_systems/makefilepackage.html https://spack.readthedocs.io/en/latest/build_systems/cmakepackage.html

https://spack-tutorial.readthedocs.io/en/latest/tutorial_buildsystems.html#makefile https://spack-tutorial.readthedocs.io/en/latest/tutorial_buildsystems.html#cmake

Conditionals based on build system https://spack.readthedocs.io/en/latest/packaging_guide.html#build-system-variant

The existing edit(), build_targets(), install() go inside class MakefileBuilder(spack.build_systems.makefile.MakefileBuilder): instead of being in the class Castep(CMakePackage, MakefilePackage): which is left with the version, variants, build_system, depends_on statements.

And we add a class CMakeBuilder(spack.build_systems.cmake.CMakeBuilder): with cmake(), build(), install() .

[heatherkellyucl]

So I stop forgetting: if after setting cmake_args nothing else unusual needs to be done in the cmake() configure step, it doesn't need to exist in the recipe.

For CASTEP, the build and install steps are not the CMakeBuilder default make and make install so they do need to exist in the recipe - they run further cmake commands instead of directly using a generated Makefile.

https://spack.readthedocs.io/en/latest/packaging_guide.html#overriding-builder-methods

[heatherkellyucl]

Apparently they are cmake's generator-independent version of the commands, so the make and make install ought to work unmodified. Changes pushed to dev/packages/castep/package.py ready to test that.

[heatherkellyucl]

I want to leave my existing myriad environment with the previous CASTEP in it alone for the moment.

Have made a copy of my existing castep module into /shared/ucl/apps/spack/0.22/hk-initial-stack/modules/additional to make sure I keep it.

spack env create --include-concrete myriad castep-cmake
spack -e castep-cmake add castep@23.1.1 +libxc

As written, this gets you

(spacksite: hk-initial-stack) [ccspapp@build01 hpc-spack]$ spack env create --include-concrete myriad castep-cmake
==> Updating view at /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/castep-cmake/.spack-env/view
==> Error: The environment view in /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/castep-cmake/.spack-env/view could not be created, because the following two specs project to the same prefix:
    hdf5@=1.14.3%gcc@=12.3.0+cxx+fortran+hl~ipo~java~map~mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make patches=82088c8 arch=linux-rhel7-cascadelake, and
    hdf5@=1.14.3%gcc@=12.3.0~cxx+fortran+hl~ipo~java~map+mpi+shared~subfiling~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make patches=82088c8 arch=linux-rhel7-cascadelake.
    To resolve this issue:
        a. use `concretization:unify:true` to ensure there is only one package per spec in the environment, or
        b. disable views with `view:false`, or
        c. create custom view projections.

because it isn't specifying the things that the yaml files do. I edited /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/castep-cmake/spack.yaml manually to add

  view: false
  concretizer:
    unify: when_possible

and the module projections.

spack -e castep-cmake concretize
spack -e castep-cmake uninstall castep
spack -e castep-cmake concretize
spack -e castep-cmake install

The uninstall will complain that other environments still have a castep so it can't be uninstalled. That seems to be ok for what I'm doing and then the last install will do a new build.

Got an error in the recipe that needs fixing!

==> Installing castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj [101/103]
==> No binary for castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj found: installing from source
==> Using cached archive: /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/cache/_source-cache/archive/8d/8d922c641c99fc6e4f5b4f7f2478abab897065850e454fb4968154ddbb566388.tar.bz2
==> No patches needed for castep
==> castep: Executing phase: 'cmake'
==> Error: KeyError: '"openmp" is not a variant of "castep"'

/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/lib/spack/spack/build_systems/cmake.py:504, in define_from_variant:
        501            variant = cmake_var.lower()
        502
        503        if variant not in self.pkg.variants:
  >>    504            raise KeyError('"{0}" is not a variant of "{1}"'.format(variant, self.pkg.name))
        505
        506        if variant not in self.pkg.spec.variants:
        507            return ""

See build log for details:
  /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-build-out.txt

[heatherkellyucl]

(I'd defined it in the cmake_args but forgotten to add it as a variant for the cmake build only).

[heatherkellyucl]

==> Installing castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj [99/103]
==> No binary for castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj found: installing from source
==> Using cached archive: /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/cache/_source-ca
che/archive/8d/8d922c641c99fc6e4f5b4f7f2478abab897065850e454fb4968154ddbb566388.tar.bz2
==> No patches needed for castep
==> castep: Executing phase: 'cmake'
==> castep: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j6'

4 errors found in build log:
     1785    cd /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23
             .1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-build-j5vyewp/Source && /lustre/shared/ucl/apps/spac
             k/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/cmake-3.27.9-iluq3vuls24uvvls4eemtkhsrk3affln/bin/cmake -D BUILDINFO_DATA_SOURCE="/lustre/shared/ucl/apps/spack/0.22/hk-
             initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spa
             ck-build-j5vyewp/buildinfo/buildinfo_data.f90" -D BUILDINFO_DATA_TEMPLATE="/lustre/shared/ucl/ap
             ps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pj
             lhyx2aa3begsj/spack-src/cmake/buildinfo_data.in" -D CASTEP_BINARY_DIR="/lustre/shared/ucl/apps/s
             pack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx
             2aa3begsj/spack-build-j5vyewp" -D CASTEP_SOURCE_DIR="/lustre/shared/ucl/apps/spack/0.22/hk-initi
             al-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-sr
             c" -D CMAKE_BUILD_TYPE="FAST" -D COMMS_ARCH="mpi" -D BLAS_TYPE="openblas" -D FFT_TYPE="fftw3" -D
              CMAKE_Fortran_COMPILER_ID="GNU" -D CMAKE_Fortran_COMPILER_VERSION="12.3.0" -D COMMS_VERSION="Open\ MPI\ v4.1.6" -D MATHLIB_VERSION="" -P /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-src/cmake/buildinfo_script.cmake
     1786    make[2]: Leaving directory '/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-build-j5vyewp'
     1787    make  -f Source/OTFG/CMakeFiles/fundamental_deps.dir/build.make Source/OTFG/CMakeFiles/fundamental_deps.dir/build
     1786    make[2]: Leaving directory '/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-build-j5vyewp'
     1787    make  -f Source/OTFG/CMakeFiles/fundamental_deps.dir/build.make Source/OTFG/CMakeFiles/fundamental_deps.dir/build
     1788    make[2]: Entering directory '/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccs
             papp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-build-j5vyewp'
     1789    [ 71%] Building Fortran object Source/OTFG/CMakeFiles/fundamental_deps.dir/otfg_data.f90.o
     1790    cd /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23
             .1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-build-j5vyewp/Source/OTFG && /lustre/shared/ucl/apps
             /spack/0.22/hk-initial-stack/spack/lib/spack/env/gcc/gfortran  -I/lustre/shared/ucl/apps/spack/0
             .22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/openmpi-4.1.6-vfg7lekutaigi
             nlgvs57titvpvnjgys5/include -I/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/openmpi-4.1.6-vfg7lekutaiginlgvs57titvpvnjgys5/lib -I/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder_
             _/__spack_path_placeholder__/__spack_path_placeholder__/linux-rhel7-cascadelake/gcc-12.3.0/fftw-
             3.3.10-cg6xo4d7umivzzbuoewey7pjr2jrca5x/include -fopenmp -J../../mod -fPIC -fconvert=big-endian
             -fbacktrace -ffpe-summary=none -ffree-form -O3 -funroll-loops -ftree-loop-distribution -g -fback
             trace -march=native -c /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/s
             pack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-src/Source/OTFG/otfg_data.f90 -o
              CMakeFiles/fundamental_deps.dir/otfg_data.f90.o
  >> 1791    CMake Error at /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-sta
             ge-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-src/cmake/buildinfo_script.cmake:18 (fil
             e):
     1792      file STRINGS file
     1793      "/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23
             .1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-src/gitinfo.dat"
     1794      cannot be read.
     1795
     1796
  >> 1797    make[2]: *** [Source/CMakeFiles/CASTEP_BUILDINFO_DATA_OBJ.dir/build.make:78: buildinfo/buildinfo_data.f90] Error 1
     1798    make[2]: Leaving directory '/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccsp
             app/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-build-j5vyewp'
  >> 1799    make[1]: *** [CMakeFiles/Makefile2:1139: Source/CMakeFiles/CASTEP_BUILDINFO_DATA_OBJ.dir/all] Error 2
     1800    make[1]: *** Waiting for unfinished jobs....
     1801    [ 71%] Building C object LibSource/libxc-5.2.2/CMakeFiles/xc.dir/src/mgga_x_mn12.c.o

...
     1927    [ 80%] Built target xc
     1928    make[1]: Leaving directory '/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-build-j5vyewp'
  >> 1929    make: *** [Makefile:169: all] Error 2

See build log for details:
  /lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-build-out.txt

"/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-src/gitinfo.dat" cannot be read.

[heatherkellyucl]

/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/build_stage/ccspapp/spack-stage-castep-23.1.1-j5vyewpzwebpsro33pjlhyx2aa3begsj/spack-src/cmake/buildinfo_script.cmake lines 14-18:

if (NOT_GIT_VC)
  #
  # Try to retrieve any saved git state from gitinfo.dat
  #
  file(STRINGS "${CASTEP_SOURCE_DIR}/gitinfo.dat" GIT_SAVED_LIST)

That file indeed doesn't exist.

[heatherkellyucl]

Need to find out if this is an issue only inside Spack or with the CASTEP cmake build process in general, since it builds some other libraries itself and I'm not sure what is trying to find gitinfo.

Trying with the Spack-built module dependencies loaded outside Spack.

module use /shared/ucl/apps/spack/0.22/hk-initial-stack/modules/lustre/shared/ucl/apps/spack/0.22/hk-initial-stack/spack/var/spack/environments/myriad/linux-rhel7-cascadelake/
module load gcc/12.3.0/gcc-11.2.1
module load cmake/3.27.9/gcc-12.3.0
module load rsync/3.2.7/gcc-12.3.0
module load openblas/0.3.26/gcc-12.3.0 
module load fftw/3.3.10-openmpi/gcc-12.3.0
module load perl/5.38.0/gcc-11.2.1
module load openmpi/4.1.6/gcc-12.3.0 
module load libxc/6.2.2/gcc-12.3.0
cd /home/ccspapp/Scratch/castep/castep-castep-65582dd54bf3/
cmake -B ../castep_2023.1.1/build -DWITH_MPI=ON -DWITH_LIBXC=ON -DCMAKE_INSTALL_PREFIX=../castep_2023.1.1 

cd ../castep_2023.1.1/build/
make
make install

[heatherkellyucl]

Got to here in make:

[ 67%] Linking Fortran static library ../../lib/libutility.a
[ 67%] Built target utility
[ 67%] Generating buildinfo_data.f90
fatal: Not a git repository (or any parent up to mount point /lustre)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
CMake Error at /home/ccspapp/Scratch/castep/castep-castep-65582dd54bf3/cmake/buildinfo_script.cmake:28 (string):
  string sub-command STRIP requires two arguments.

make[2]: *** [Source/CMakeFiles/CASTEP_BUILDINFO_DATA_OBJ.dir/build.make:75: buildinfo/buildinfo_data.f90] Error 1
make[1]: *** [CMakeFiles/Makefile2:1136: Source/CMakeFiles/CASTEP_BUILDINFO_DATA_OBJ.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

[heatherkellyucl]

So the same buildinfo_script.cmake but a different branch, it didn't go into the if (NOT_GIT_VC) part, instead into the else further down.

Will see if not cding is any different.