Optimum `GPU/CPU` overlapping and `GPU` performance heavily rely on using large `MPI` ra##nks

[koarakawaii]

Issue

GAMER GPU test on new cluster (caler) suggests that performance (validated using Perf_Overall in Record__Performance) keeps increasing without saturation when MPI ranks increasing. The best performance is achieved by 16 MPI ranks, while using 32 MPI ranks will exceed GPU memory limitation. The time-step-averaged Perf_Overall summary is given as below (TIMING_SOLVER off)

1 MPI rank , GPU_NSTREAM = -1 (16): 4.3045e+07
2 MPI ranks, GPU_NSTREAM = -1 (16): 5.6315e+07
4 MPI ranks, GPU_NSTREAM = 10     : 6.4590e+07
8 MPI ranks, GPU_NSTREAM =  4     : 6.9180e+07
16 MPI ranks, GPU_NSTREAM = 1     : 7.3055e+07

Performance test

• ClusterMerger problem with patch number further increased (by adjusting the AMR refinement criteria), compared to the default test problem setting
  • Simulation type: FLUID + PARTICLE
  • Total time step: 20
  • NX0_TOT_X: 128; NX0_TOT_Y: 128; NX0_TOT_Z: 128
  • MAX_LEVEL: 3
  • AMR criteria: Input__Flag_Rho, Input__Flag_NParPatch and Input_Flag_Lohner(Dens). Content for each file is shown below:
    • Input__Flag_Rho

      # Level   Number of Particles per Patch
      0                              0.0
      1                              1.e5
      2                              1.e5

    • Input__Flag_NParPatch

      # Level   Number of Particles per Patch
      0                               0
      1                              20
      2                              40

    • Input__Flag_Lohner

      # Level    Threshold_Refine  Threshold_Derefine              Filter              Soften          MinDensity
      0                  0.80                0.80                0.01                0.00             1.50e-2
      1                  0.80                0.80                0.01                0.00             1.50e-2
      2                  0.80                0.80                0.01                0.00             1.50e-2