amg2023 is not utilizing all ranks

[jdomke]

likely root cause: https://github.com/LLNL/benchpark/blob/27aafcd4d9849fc6671a521a9b34fd79c9cf4b39/experiments/amg2023/openmp/ramble.yaml#L27

run file:

⇒  cat workspace/amg2023/openmp/RCCS-Fugaku-Fujitsu-A64FX-TofuD/workspace/experiments/amg2023/problem1/amg2023_omp_problem1_2_
8_2_2_2_55_55_55/execute_experiment
#!/bin/bash
# Copyright 2023 Lawrence Livermore National Security, LLC and other
# Benchpark Project Developers. See the top-level COPYRIGHT file for details.
#  
# SPDX-License-Identifier: Apache-2.0

#PJM -L "node=2"
#PJM --mpi proc=8
#PJM -L "elapse=02:00:00" -x PJM_LLIO_GFSCACHE="/vol0002:/vol0003:/vol0004:/vol0005:/vol0006"

cd /vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/workspace/amg2023/openmp/RCCS-Fugaku-Fujitsu-A64FX-TofuD/workspace/experiments/amg2023/problem1/amg2023_omp_problem1_2_8_2_2_2_55_55_55

rm -f "/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/workspace/amg2023/openmp/RCCS-Fugaku-Fujitsu-A64FX-TofuD/workspace/experiments/amg2023/problem1/amg2023_omp_problem1_2_8_2_2_2_55_55_55/amg2023_omp_problem1_2_8_2_2_2_55_55_55.out"
touch "/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/workspace/amg2023/openmp/RCCS-Fugaku-Fujitsu-A64FX-TofuD/workspace/experiments/amg2023/problem1/amg2023_omp_problem1_2_8_2_2_2_55_55_55/amg2023_omp_problem1_2_8_2_2_2_55_55_55.out"
export OMP_NUM_THREADS="8";
. /vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/workspace/spack/share/spack/setup-env.sh
spack env activate /vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/workspace/amg2023/openmp/RCCS-Fugaku-Fujitsu-A64FX-TofuD/workspace/software/amg2023-omp.problem1
mpiexec amg -P 2 2 2 -n 55 55 55 -problem 1 -keepT >> "/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/workspace/amg2023/openmp/RCCS-Fugaku-Fujitsu-A64FX-TofuD/workspace/experiments/amg2023/problem1/amg2023_omp_problem1_2_8_2_2_2_55_55_55/amg2023_omp_problem1_2_8_2_2_2_55_55_55.out"

• 2 nodes and 8 ppn => 16 ranks
• mpiexec amg -P 2 2 2 => only running on 8 ranks