Install Request: ORCA 6.0

[kaibinary]

Request #1284496

Link: https://orcaforum.kofo.mpg.de/app.php/portal

Cluster: Young

ORCA 6.0 was released last month and is available via Spack. Can potentially be released as part of https://github.com/UCL-ARC/hpc-spack/issues/44. Related request: https://github.com/UCL-RITS/rcps-buildscripts/issues/474

[kaibinary]

New requests: Request #1294756 Request #1337718

[heatherkellyucl]

(The below has been tested up until module file generation).

Orca 6.0 requires OpenMPI 4.1.6. This is the OpenMPI we're having in our Spack software stack once we release that.

In the meantime, Spack can be used to give a personal install of that OpenMPI as follows (using https://github.com/UCL-ARC/hpc-spack/wiki/Personal-Spack-install-on-cluster-(non%E2%80%90spacksites))

# git clone a stable release of spack (spack recommends these manyFiles and depth settings)
git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git --branch v0.22.2

Set up a load_scls.sh script to give you a newer default gcc 11 and python 3 for Spack to start with:

#!/bin/bash
# Source this.

UCL_SCLS="rh-python38 devtoolset-11"

for a in ${UCL_SCLS}
do
   source /opt/rh/${a}/enable
done
export X_SCLS="${UCL_SCLS} ${X_SCLS}"

# setup Spack
source load_scls.sh
# this sets $SPACK_ROOT for you
source spack/share/spack/setup-env.sh

Spack's default config.yaml is $SPACK_ROOT/etc/spack/defaults/config.yaml. Things you may wish to change:

  build_stage:
    - $tempdir/$user/spack-stage
    - $user_cache_path/stage

  # build_jobs: 16

build_stage is where all the temporary building happens and must not be in /tmp. (Leaving this setting as is and setting export TMP=$XDG_RUNTIME_DIR would use a per-user temporary space that gets cleared out when you log out - you may wish these to be in Scratch if you want to be able to see them later).

build_jobs is how parallelised it will do its builds - don't make this too high on the login nodes. I recommend not larger than 4. If you build in a job you can make this larger or leave it unset.

You can also export SPACK_USER_CACHE_PATH= to change where $user_cache_path refers to. By default this is in ~/.spack

SPACK_USER_CACHE_PATH: Override the default path to use for user data (misc_cache, tests, reports, etc.)

You can then start to use Spack.

# check what compilers spack sees (anything not gcc 11 will depend on what modules you have loaded)
 spack compilers
==> Available compilers
-- gcc rhel7-x86_64 ---------------------------------------------
gcc@4.9.2  gcc@4.8.5  gcc@11.2.1

# install openmpi 4.1.6 using Spack and gcc 11
# This makes it SGE-aware so that it doesn't need telling where a machinefile is to find out what nodes
# it can run on.
# It also removes fabrics that we don't have.

spack install openmpi@4.1.6%gcc@11.2.1 fabrics=psm2,cma,ucx,ofi schedulers=sge

This will build many dependencies of openmpi and finally openmpi itself.

You will then be able to activate spack again and spack load openmpi@4.1.6 and can also get Spack to make a module for it (to be added next, this is recommended for use in jobs as spack load takes longer).

If you get this error:

==> Error: The "config" section of the configuration needs to be written to disk, but is currently using a deprecated format. Please update it using:

        spack config [--scope=<scope>] update config

Note that previous versions of Spack will not be able to use the updated configuration.

you've got a ~/.spack from an older version of Spack and may need to rename/delete it.

Modules:

spack module tcl refresh: create TCL modules for all currently installed packages

spack config add modules:default:enable:[tcl]: turn on TCL module creation for all future packages.

Spack modules won't set LIBRARY_PATH or LD_LIBRARY_PATH by default in its modules and you are likely to need this when using libraries with external-to-Spack programs. You can tell Spack to add this for specific packages by editing ~/.spack/modules.yaml like this: https://spack.readthedocs.io/en/latest/module_file_support.html#setting-environment-variables-for-selected-packages-in-config

Or for all packages by editing ~/.spack/modules.yaml like this:

  modules:

    prefix_inspections:
      lib: ["LD_LIBRARY_PATH", "LIBRARY_PATH"]
      lib64: ["LD_LIBRARY_PATH", "LIBRARY_PATH"]