Benchmarking

[gawrysz]

Integration of the benchmarking_mergeable branch that should allow for taking an up-to-date performance/scalability profile of Piernik on a single machine.

[gawrysz]

An example of comparison made on single CPU, varying code version, compiler and MPI library: This test was taken on a desktop CPU (Ryzen 7 (8c/16t), clocked at 3900 MHz – non-standard setting, with all "turbo" disabled for better consistency, running, with 2-channel RAM, on up-to-date Fedora 37 with gfortran 12.2.1). We can see here that:

• The currently available MPICH 4.0.2 and OpenMPI 4.1.4, available in Fedora 37 does not differ much in performance: the red line is placed nearly on top of the orange one.
• We have gained a lot of performance in the last 7 years: depending on test, the blue line is placed from slightly to significantly higher than the orange/red one. This came at the cost of longer compilation time ;-)
• The performance of the oneAPI Intel's compiler (ifx 2023.1.0) paired with Intel's MPI is far from impressive. It produces code ranging from from slightly slower in the hydro solver to terribly slow in the multipole solver and multigrid anisotropic diffussion. There is some use of the Intel tools, but seems to be quite limited. It can't even serve well for profiling and optimization, at least on non-Intel CPUs.

Similar tests can be made on a server-class Intel-based machine (dual Intel Xeon Silver 4316 (2× 20c/40t), with default clock settings and 8-channel RAM on each CPU). This one was run on an updated Fedora 32:

• The pure hydro test exhibits some performance gains of ifx over gfortran, mostly for single thread and HT regime (typically not interesting from HPC point of view).
• Multipole and multigrid performance of ifx is still inferior, compared to gfortran.
• The increasing execution time of weak scaling crtest seen on gfortran tests for HT-regime is something peculiar to this CPU family (I haven't seen it on Epyc CPUs), and is limited to one MPI communication routine for internal boundaries exchange (can't remember which one, but we don't rely on HT in HPC anyway).

Please note that the transition to HT-regime (41-80 threads on dual Xeon) or SMT-regime (9-16 threads on Ryzen) is always associated with performance decrease by a factor up to 2. When it is significantly lower than 2 it perhaps means that there is some room for optimization by achieving higher usage of internal execution units in the CPU core. But the gain is not guaranteed on current and future CPUs, because if one can reach high Instruction-Per-Cycle ratio in the code, the CPU can readily lower its clock due to Power/Current/Temperature constraints.