kennyroche
commented
4 years ago I am interested to talk to you about this work you did. I really appreciate this note. It’s a lot of hard work to get all this information collected and written up simply and clearly as it is here. “It is hardly obvious from this output what one should do to improve performance of the code.” This is the key statement, and there are rarely obvious answers. Whoever wrote this note,I assume Brian, I am curious to talk to you about these results and how / if it is possible to demystify the measurements of code execution. I have no funding for this this moment, but can easily justify discussing the issue of tools and dissecting performance of computer experiments on certain architectures / platforms. Regards, -k
bcfriesen
cmahrens
jmohdyusof
rvinaybharadwaj
Background
A goal for ExaRL is to design a performance assessment framework that can be used to track performance of the code over time and during its development. This can be a challenging task for ExaRL due to its reliance on TensorFlow, which is a complex framework that in turn relies on other complex GPU accelerated libraries like cuDNN.
The standard approach of profiling the code as a 'black box' using a tool like NVIDIA Nsight Compute or Nsight Systems does not yield very useful results when the code relies heavily on TensorFlow, which launches millions or billions of GPU kernels in a typical run, and any of those kernels are highly tuned in libraries like NVIDIA cuDNN or Eigen. An example of this problem, drawn from a TensorFlow example code for classifying images, is shown below:
The GPU kernel statistics as reported by Nsight Systems are shown below:
It is hardly obvious from this output what one should do to improve performance of the code.
So we must adjust this approach in order to be able to collect actionable performance data about ExaRL. Most likely, we will need to use a combination of 'black box' profiling tools and domain-specific tools which have some awareness of what kinds of calculations the code is doing.
Collecting performance data
We can use a combination of performance analysis tools to understand the overall performance characteristics of the code. A few components and proposals are described below.
Timers
These are simple to implement and can summarize strong and weak scaling behavior easily.
MPI analysis
ExaRL uses MPI for inter-node communication. MPI performance is straightforward to measure, including characteristics like time spent at barriers, load imbalance, etc. Many tools can measure these quantities via sampling of each MPI task; this activity has low overhead, and can typically be used for even high-MPI-concurrency runs. Open source tools like TAU and HPCToolkit can be used for this, along with several other proprietary tools like Arm MAP. So far we have been using TAU on the Cori GPU cluster at NERSC, with reasonably good results. The
jumpshotGUI shows results like the following:and the
pprofanalysis tool shows quantitative results:GPU performance
Nsight Systems can be useful for understanding data movement between CPU <-> GPU during the run, and also for gaining insight into any GPU kernels which are not part of TensorFlow. It also uses sampling to collect data, and thus can be used as a 'black box' profiling tool with relatively low overhead; since a typical ExaRL calculation is quite long, it is best to disable CPU sampling in Nsight Systems by adding the
-s noneflag; otherwise, the resulting profile will be enormous and will take hours to process.An example profile using Nsight Systems is shown below. In this case, it looks like the majority of GPU activity is spent executing TDLG kernels, in which case Nsight Compute could possibly be used to improve performance.
Nsight Compute can be useful for tuning hand-written kernels, like LibTDLG, but it is much less useful when the runtime is dominated by TensorFlow activity.
TensorFlow performance
TensorFlow includes its own profiling framework which, unlike 'black box' profiling tools, has significant domain-specific awareness about what the calculation is doing. It is likely we will need to rely on this to supplement the above tools if the goal is to improve the performance of the TensorFlow portions of ExaRL.
Representing and tracking performance data
It will be useful to track the performance characteristics of ExaRL as the code base develops. One way to do this is to integrate ExaRL into the ECP GitLab continuous integration infrastructure which is already available at NERSC. We could configure a GitLab runner to launch once per day, check out the latest version of the code, run it, and store the performance characteristics of the code in a file or database that is then visualized on a website. The website can be hosted in Spin.
Some characteristics like execution time, or fraction of time spent in MPI, can be represented simply with timings, and plotted on a graph as a function of git commit or date, like here. Other characteristics which may have more complex information, like the output from Nsight Systems or Nsight Compute or TensorFlow Profiler, may require a different approach for visualizing the data as a function of time.