[CSD3] Update configurations and Spack environments

[mirenradia]

This PR updates the ReFrame configurations for CSD3 and the corresponding Spack environments. Furthermore it also adds the new Sapphire Rapids partition.

Some key changes are explained below.

Operating System

On CSD3, we have 2 different OSs:

• CentOS 7 - a rebuild of Red Hat Enterprise Linux (RHEL) 7
• Rocky Linux 8 - a rebuild of RHEL 8

The Cascade Lake partition run on CentOS 7 and all other partitions (i.e. Ice Lakes and Sapphire Rapids) run on Rocky Linux 8. In order to avoid bugs and library incompatibilities, it is important to use the software stack built for the correct OS. One of the main problems with the current csd3-icelake Spack environment is that there are lots of external packages built for CentOS 7/RHEL 7 and as a result I experienced segfaults on the example Sombrero benchmark with the default Spack spec. I have therefore completely recreated it from scratch and only included packages built for Rocky Linux 8.

In order to stress the different OSs to users of this repo on CSD3, I have renamed the existing systems as follows

• csd3-cascadelake/compute-node -> csd3-centos7/cascadelake
• csd3-icelake/compute-node -> csd3-rocky8/icelake

Note that there are two sets of login nodes login-p-[0-4] that are running Cent OS 7 and login-q-[0-4] that are running Rocky Linux 8.

Sapphire Rapids Partition

This currently uses the same software stack and OS as the Ice Lakes hence the common compilers.yaml and packages.yaml for the Spack environment. The only difference in the Spack environments is the specified target.

The Sapphire Rapids partition has been added as a new partition (in the ReFrame sense) under the csd3-rocky8 system.

Preferred compilers and MPI implementation

I have set the preferred compiler to intel for all Spack environments and the preferred MPI implementation to intel-mpi (for CentOS 7) or intel-oneapi-mpi (for Rocky Linux 8). These are both in the default modules that are loaded on CSD3 and we generally observe better performance than with gcc/openmpi.

Reference values

I have updated some of the existing reference values for the benchmarks.

I have tested this PR with the following benchmarks:

• grid
• hpl
• ramses
• sombrero
• sphng
• swift
• trove-pdsyev
• trove

There are still a few minor changes I wish to make before asking for a review hence why I'm making this a draft PR for now.

[mirenradia]

I've rebased the changes onto main. This is now ready to be merged.

[mirenradia]

Can you please sort alphabetically packages in the spack configurations? That makes easier to see what's in there.

It would be nice if spack external find did this automatically. In any case, I've done it now (by hand :see_no_evil:).