Allow user to configure the analyses they want to see

[rsdunlapiv]

Given an application trace, there are many potential analyses possible, so the user needs to be able to specify what analyses they want to see, i.e., what plots should be generated. Due to the size of the traces, we cannot generate a single JSON representation and expect the browser to process it. Instead, each specific analysis will need a custom JSON stream (targeted for a specific javascript plot). For example, the user could specify that they are interested in looking at load balance and would like a set of load balance plots for the first two levels of the timing tree -- and nothing deeper.

An appealing option for configuration is through command line arguments:

$ ./esmf-profiler --load-balance=depth:2

From this you would expect a series of load balance plots (stacked bar charts). One for the first level. And one for each of the second levels in the tree. The depth parameter would allow the user to control the tradeoff between how much time it takes to generate the profile plots and how much detail they see.

The user could know ahead of time they they are interested only in specific regions in the timing tree, such as the region called "[ATM-TO-OCN] RunPhase1". This calls for a filter option:

$ ./esmf-profiler --load-balance=filter:"[ATM-TO-OCN] RunPhase1"

In this case the profiler would generate a bar chart showing execution time for every PET for the region named "[ATM-TO-OCN] RunPhase1". (i.e., PET number on the x-axis, total execution time on the y-axis).

This could be generalized to a regex style filter:

$ ./esmf-profiler --load-balance=filter:"*ATM-TO-OCN*"

In this case you would expect to see a set of bar charts, one for each timed region with a name containing the substring "ATM-TO-OCN".

The user might also want to restrict the set of PETs shown in the load balance plot:

$ ./esmf-profiler --load-balance=pets:0-155,filter:"[ATM-TO-OCN] RunPhase1"

This would show a load balance bar chart for the region named "[ATM-TO-OCN] RunPhase1" with only PETs 0-155 in the x axis.

Longer term, additional analyses will be possible and could also be controlled using command line arguments. One example is showing, for a single PET, the execution time of each iteration of a give region (see #3). This requires some information from the user including which PET and the region name. For example:

$ ./esmf-profiler --execution-time=pet:124,name:"[ATM] RunPhase1"

This would generate a plot as shown in #3. This analysis requires capturing the individual RegionEnter and RegionExit events.

Another example of this is an analysis that shows memory allocations over time. (NOTE: We do not yet have these events in the trace, but it can be extended to support this.)

# example of how to specify a memory plot over time showing Resident Set Size in the y axis for PET 214. 
$ ./esmf-profiler --memory=type:rss,pet=214

The various options could be combined:

# generate a load balance bar chart for depth 1 (includes all PETs) AND a memory usage chart for PET 0 showing the High Water Mark
$ ./esmf-profiler  --load-balance=depth:1 --memory=type:hwm,pet=0

Using a dashboard-like layout, each of the requested analysis could be linked in a menu on the left. Then the user could click on each analysis and be shown the corresponding plot(s).

[rsdunlapiv]

The user also needs to point the tool to the directory containing the binary (CTF) trace. So the commands above would have an additional argument such as:

$ ./esmf-profiler --load-balance=depth:4 /path/to/trace/directory

[rsdunlapiv]

Error checking will be needed:

$ ./esmf-profiler --memory=type:rss,pet:16  /path/to/my/tracedir

ERROR: A memory profile was requested for PET 16, but there is no memory information in the trace.

[rsdunlapiv]

@rlongio any thoughts on this?

[ryanlong1004]

Sorry for the delay.

Yes, these can all be implemented in the user layer rapidly once the backend logic is finished. It has an API that will allow the user to specify the data they need (filters, etc.) and how many layers depth to go, so if there is a cookie cutter high chart you'd like to use we can get something visually together quicker.

On a side note, after working with a larger binary trace, I did some profiling as the execution time was near 30 minutes. I was able to get that down to less than 5, but it required some refactoring (ie time in bt2 library is expensive).

I expect to finish this by today, or at least have a working MVP.

Regardless, this week I'm going to start sticking static data in the front end to explore some visuals.

The requested features will be either in the first iteration or will be put in as tasks for the second.

[rsdunlapiv]

Sounds good. Good work reducing the time. Some of the time in bt2 is just an artifact of file system access, so we may have limited options in optimizing that side, especially for large traces. However, that is not a showstopper as the user will have the expectation of waiting to post-process large traces.

For your first iteration, you are essentially delivering this, even if the CLI is not set up:

$ ./esmf-profiler --load-balance=depth:2  /path/to/trace

[ryanlong1004]

@rsdunlapiv Can we close this issue?

[rsdunlapiv]

We need to leave this one open since we have not yet implemented command line parameters for the user to define the analyses they want. But it is not part of a MVP.