Add UKAEA NESO benchmark

[tkoskela]

This would be beneficial for the AQUIFER project.

NESO is a combination of packages that all build with spack. Their packages are not in upstream spack, but in a local repo

The developers told me that

• You don't necessarily need to use their spack environment, its main aim to ease setup for developers
• Most problems will come from building nektar++ (one of the dependencies), once you have that you are mostly done
• @will-saunders-ukaea and @jwscook are happy to help with issues

We can use the Electrostatic2D3V and SimpleSOL examples to set up benchmarks.

Next steps

• [ ] Update neso-Spack to 0.21.0
• [ ] Build neso.nektar++ and neso.neso-particles using upstream spack packages
• [ ] Test spack build on a HPC system [Archer2, Csd3, Isambard?]
• [ ] Figure out a way to add a new repo from a git repository into the ReFrame workflow
• [ ] Test spack build through ReFrame

[tkoskela]

I have successfully built neso-particles in a fresh spack environment with (approximately)

git clone git@github.com:ExCALIBUR-NEPTUNE/NESO.git
git submodule update --init
spack repo add NESO/NESO-spack
spack env create --without-view -d <path-to-spack-env>
spacktivate <path-to-spack-env>
spack compiler find
spack external find [relevant packages]
spack add neso.neso-particles@0.1.0%gcc@11.3.0
spack install

==> Installing neso-particles-0.1.0-s6vqg4n2xcmyjd4yaavfx2fwuznbdagx
==> No binary for neso-particles-0.1.0-s6vqg4n2xcmyjd4yaavfx2fwuznbdagx found: installing from source
==> No patches needed for neso-particles
==> neso-particles: Executing phase: 'cmake'
==> neso-particles: Executing phase: 'build'
==> neso-particles: Executing phase: 'install'
==> neso-particles: Successfully installed neso-particles-0.1.0-s6vqg4n2xcmyjd4yaavfx2fwuznbdagx
  Fetch: 2.97s.  Build: 2m 26.85s.  Total: 2m 29.82s.

Notes

• Their spack recipes require spack 0.19.0. I had problems with the spack 0.21.0 development branch I had on my machine
• I was getting compile errors in nektar++ with the gcc@11.4.0 that was installed on my Ubuntu 22.04 machine. According to Will:

  I was getting similar errors building with gcc 11.4 (apt installed on Ubuntu 22.04). 11.3 (spack installed) works fine for me. I think this issue might be the root cause.

spack installing gcc@11.3.0 and building with it also worked for me.

• The python executable must be in $PATH to install neso-particles. On my machine I only had python3 in the global environment, which didn't work. I did the install in a python venv to get around this.

[matt-graham]

Just to check is the plan for this to add a benchmark specifically for NESO-particles (the library for enabling particle transport over unstructured meshes) or NESO as a whole (which combines NESO-particles with Nektar++ and exposes several solvers for doing particle-in-cell like simulations)? The latter seems like it might be easier to structure into a benchmark as there are already various example configurations for the existing solvers that could be used as the basis for a benchmark.

[will-saunders-ukaea]

There is an "in progress" benchmark solver [1] in a branch of NESO. Unfortunately I started this as part of some work to improve the projection/evaluation operations (I needed a way to actually assess the changes). That projection/evaluation work is quite far from being ready to merge into main.

In NESO main - currently there is: Electrostatic2D3V - 2D fairly uniform distribution of work, mostly particle projection/evaluation little FEM work. SimpleSOL - with particles is very non-uniform work as they are restricted to the ends of the domain. H3LAPD - solver this is a more representative case from the science side (and has fairly uniform work across particles and FEM). @oparry-ukaea is getting this into main currently.

In NESO-Particles - I have advection benchmarks for development that could be made more user friendly - uniform work, easier to install. But much less scientific relevance.

[1] https://github.com/ExCALIBUR-NEPTUNE/NESO/tree/expr/expansion-code-gen/solvers/Benchmark

[tkoskela]

Ideally I think it would be useful to have both NESO and NESO-particles benchmarks, but in the interest of ensuring we deliver something, it looks like a good idea to start with the examples in NESO main and develop a whole NESO benchmark.