Closed mirenradia closed 8 months ago
I've rebased the changes onto main
. This is now ready to be merged.
Can you please sort alphabetically packages in the spack configurations? That makes easier to see what's in there.
It would be nice if spack external find
did this automatically. In any case, I've done it now (by hand :see_no_evil:).
This PR updates the ReFrame configurations for CSD3 and the corresponding Spack environments. Furthermore it also adds the new Sapphire Rapids partition.
Some key changes are explained below.
Operating System
On CSD3, we have 2 different OSs:
The Cascade Lake partition run on CentOS 7 and all other partitions (i.e. Ice Lakes and Sapphire Rapids) run on Rocky Linux 8. In order to avoid bugs and library incompatibilities, it is important to use the software stack built for the correct OS. One of the main problems with the current
csd3-icelake
Spack environment is that there are lots of external packages built for CentOS 7/RHEL 7 and as a result I experienced segfaults on the example Sombrero benchmark with the default Spack spec. I have therefore completely recreated it from scratch and only included packages built for Rocky Linux 8.In order to stress the different OSs to users of this repo on CSD3, I have renamed the existing systems as follows
csd3-cascadelake/compute-node
->csd3-centos7/cascadelake
csd3-icelake/compute-node
->csd3-rocky8/icelake
Note that there are two sets of login nodes
login-p-[0-4]
that are running Cent OS 7 andlogin-q-[0-4]
that are running Rocky Linux 8.Sapphire Rapids Partition
This currently uses the same software stack and OS as the Ice Lakes hence the common
compilers.yaml
andpackages.yaml
for the Spack environment. The only difference in the Spack environments is the specified target.The Sapphire Rapids partition has been added as a new partition (in the ReFrame sense) under the
csd3-rocky8
system.Preferred compilers and MPI implementation
I have set the preferred compiler to
intel
for all Spack environments and the preferred MPI implementation tointel-mpi
(for CentOS 7) orintel-oneapi-mpi
(for Rocky Linux 8). These are both in the default modules that are loaded on CSD3 and we generally observe better performance than withgcc
/openmpi
.Reference values
I have updated some of the existing reference values for the benchmarks.
I have tested this PR with the following benchmarks:
There are still a few minor changes I wish to make before asking for a review hence why I'm making this a draft PR for now.