[DISCUSSION] Profiling GCHP Performance using external utilities

[lizziel]

Discussed in https://github.com/geoschem/GCHP/discussions/142

^{Originally posted by **WilliamDowns** May 11, 2021} This issue is intended for discussing profiling GCHP performance using various external utilities such as TAU, IPM, Intel VTune, etc. In my testing I found that using TAU causes GCHP to hang during ExtData initialization when using the `-ebs -ebs_resolution=function` option with `tau_exec`. I've successfully completed runs using `mpirun -np ${SLURM_NPROCS} tau_exec ./gchp >> ${log}`, which outputs profiling data at the MPI call level. I'm unable to compile GCHP with TAU, as CMake hangs when I set my Fortran compiler variables to TAU's requested compiler script (whether through setting `FC=$(spack location -i tau)/bin/tau_f90.sh` or using `DCMAKE_Fortran_COMPILER=$(spack location -i tau)/bin/tau_f90.sh`). My system + run config: - AWS ParallelCluster 2.10.1, centos7 - GNU Compilers 9.3.0 - Intel MPI 2019U10 - TAU 2.30.1 - GCHP 13.0.2 - c90 - 9 nodes / 324 cores (8 model nodes and 1 output node) - Emissions, Metrics, SpeciesConc diagnostics - 1 week simulation I also found that IPM does not compile with GCC 9.3.0, but does compile when using ifort 2021.2.

[lizziel]

We are switching from GitHub Discussions to GitHub Issues. I am going to delete the Discussion now. The history of the thread is as follows:

@lizziel:

@mathomp4, @tclune: could you comment on what tools you currently use to profile GEOSgcm / GEOSctm performance?

@mathomp4:

I'm going to ping @aoloso here as well as he's done more profiling recently compared to me. I think the last few times I did so was with ARM MAP and Intel Vtune. TAU has always been my archenemy. You get it working and then the next day it doesn't work.

@lizziel:

@mathomp4, did you hear back form anyone about tools recently used for profiling GEOS?

@mathomp4:

@lizziel As I said, MAP, VTune and TAU are the three tools I and @aoloso have used for GEOS. I tend to use MAP first as it is usually the most user friendly, though it can't get the detail that TAU does.