Organize Views in folders

[ebugden]

⚠️Note: In this issue, the term "view" should have been used instead of "analysis" (see comment). The issue title and description are updated, but all comments below are unchanged.

---

Organizing views analyses in folders gives users a higher-level overview of the views analyses that are offered.

Important: Only organize views in folders if it is possible to sort views into mutually exclusive categories (that are meaningful for the user). If mutually exclusive categories don't make sense for views, other organization options should be considered (ex. Filtering by tag #158).

• Ideally stick to just one folder level deep.
• Categories should be meaningful for the user. Consider if it makes more sense to group views by analysis subject (ex. Cpu, memory, disk) or by analysis purpose (ex. Latency, overhead, throughput) or something else.
• Consider in what order the folders should be presented. Alphabetical order? The order in which they’re typically used (ex. more generally applicable to more specialized/niche) to suggest a certain methodology? Some other order? For example, when sending a new message Facebook messenger starts by listing a few frequently contacted friends at the top and switches to alphabetical order for the rest of the contacts.

---

Current list of views:

[ebugden]

List of Trace Compass analyses for organizing (not exhaustive, but hopefully representative):

• Active thread
• Context CallStacks
• Context switch
• Counters
• CPU usage
• Critical path aggregated
• Debug info
• Event matching latency
• Example state system module
• Futex contention analysis
• Input/Output
• IO
• IRQ analysis
• Kernel memory usage
• Linux kernel (Control flow view?, Resources view?)
• LTTng-UST CallStack
• OS execution graph
• ROS message flow
• Sched wakeup switch latency
• Scripted analyses
• Source code assistant
• Statistics
• System call latency
• Test a builtin XML module file
• Test other analysis
• Ust memory

External analyses

• Event fields count

[ebugden]

Windows Performance Analyzer (WPA) categories:

• System activity (ex. Generic events, memory utilization)
• Computation (ex. CPU usage, DPC (Direct Procedure Call) / ISR)
• Storage (ex. Disk usage)
• Memory
• Power

[ebugden]

Perfetto trace config, probe categories and contents (subset):

• CPU
  • Coarse CPU usage counter
  • Scheduling details
  • CPU frequency and idle states
  • Syscalls
• GPU
  • GPU frequency
  • GPU memory
• Power
  • Battery drain
  • Board voltages & frequencies
• Memory
  • Native heap profiling
  • Java heap dumps
  • Kernel meminfo
  • High-frequency memory events
  • Low memory killer
  • Per process stats
  • Virtual memory stats

[ssmagula]

Hello all, Just a quick note to say what Erica outlined here makes a lot of sense to me. Grouping analyses (by "type" today and perhaps by "type of problem it solves" tomorrow?) will make analyses easier to discover, retrieve, and remember. This has to do with Miller's theory about short-term working memory ("we can recall 7 plus or minus 2"). Miller may have been a bit optimistic (see: https://www.nytimes.com/2013/09/10/science/seven-isnt-the-magic-number-for-short-term-memory.html) so anything we do to categorize, group, chunk information (while retaining power/optionality) will yield positive results.

I've been preoccupied with the winter storm in Austin week. We lost power, heat, and our house partly flooded. Now we have no water and neither do our neighbors. It's a mess. I hope to be back soon.

--Stefan

On Thu, Feb 18, 2021 at 6:07 PM ebugden notifications@github.com wrote:

Perfetto trace config, probe categories and contents (subset):

• CPU
  • Coarse CPU usage counter
  • Scheduling details
  • CPU frequency and idle states
  • Syscalls
• GPU
  • GPU frequency
  • GPU memory
• Power
  • Battery drain
  • Board voltages & frequencies
• Memory
  • Native heap profiling
  • Java heap dumps
  • Kernel meminfo
  • High-frequency memory events
  • Low memory killer
  • Per process stats
  • Virtual memory stats

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/theia-ide/theia-trace-extension/issues/157#issuecomment-781717589, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIOFFVTV5TWRP2NAX37BGLS7WTVVANCNFSM4SPZZKOQ .

[ebugden]

@MatthewKhouzam Could you sanity check this rough sort attempt?

Right now we're just trying to determine whether analyses can be sorted into useful and mutually exclusive categories? (Otherwise it may make more sense to use tags #158)

Sort attempt (with very limited knowledge of analyses):

• System activity
  • Active thread?
  • Critical path aggregated
  • Context CallStacks?
  • LTTng-UST CallStack?
  • OS execution graph?
• CPU/Computation
  • Active thread?
  • Context switch
  • Counters (performance counters?)
  • CPU usage
  • IRQ analysis
  • Sched wakeup switch latency?
  • System call latency?
• Storage
  • Input/Output (For disk or network?)
  • IO (For disk or network?)
• Memory
  • Futex contention analysis
  • Kernel memory usage
  • Ust memory

Other categories that could be relevant:

• Locking
  • Futex contention analysis
• Latency
  • Event matching latency
  • Sched wakeup switch latency
  • System call latency
• Resources
• Network
• Power
• General, Other

[MatthewKhouzam]

I am thinking... and I don't know if it's correct or not, but would it be more interesting to have several "layouts"

simple: Problem to solve e.g.

• why is my program slow?
• why is my system slow?

Advanced:

• Power:

  • power draw
  • frequency
  • temperature

• CPU Usage:

  • resources?
  • CPU Usage?

• GPU:

  • Arnaud?

• Memory:

  • Usage
  • Deadlocks maybe?

• I/O

  • Disk
  • Network

• Latency

  • examples of latencies?

Then dev mode can be more reflective of our architecture.

Thoughts?

[ebugden]

Some related questions:

• Which analyses visualize information (select, present)? vs. which transform (do calculations) and then visualize information (select, transform, present)?
• Should "analyses" that simply visualize information simply be called "visualisations"?

[ebugden]

MatthewKhouzam: would it be more interesting to have several "layouts"?

Yes, but we're starting with something more simple before working up to that. The table below shows organization steps with increasing levels of accessibility.	Lego organization	TC analysis organization
Sorting pieces by type	Grouping analyses by type #157
Labelling piece boxes with recommended usage information	Good analysis titles and usage explanations
Providing instruction sheets for specific builds	Providing problem-based analysis layouts

Here's my plan:

1. Make expert's life easier: Because a lot of current users already know how to do trace analysis, my intuition was to start by making the analyses feel like well organized lego—super easy to quickly find what you're looking for. An expert lego builder already knows what piece they need and how to use it, they just need to find it fast.
2. Build defaults based on expert's layouts: I don't have the expertise to make relevant problem-based layouts in a time efficient way, but once the experts have been "playing with their lego" and building analyses layouts for a while we could ask them for their favourite layouts for different problems (Related: #267). What's great about asking for layouts is that there's no proprietary data involved!

If we try to jump directly to problem-based layouts, we risk them not being representative or useful.

[ebugden]

Current focus: Can analyses be sorted into useful and mutually exclusive categories (i.e. folders #157)? (Otherwise it may make more sense to use tags #158)

[mirsky-work]

I tend towards (not too many) mutually exclusive categories, that will help users who already know what they are looking for, to find it quickly. Something along the lines of:

• CPU
• GPU (perhaps can be combined with CPU into "Compute")
• Memory
• Power
• Structural?
  • Callstack
  • Critical path
  • ...

[ebugden]

mirsky-work: I tend towards (not too many) mutually exclusive categories (CPU, (GPU), Memory, Power, Structural?)

I also think having around 5 categories is a good first step.

I'm going to looking through the Trace Compass User Guide to get a bit more info for a slightly better sort proposal than above, but then I'm going to need someone with more experience to review like @MatthewKhouzam.

[ebugden]

I've been organizing the wrong items! I've been organizing Trace Compass Analyses when I should have been organizing Trace Compass Views.

• The "Analyses" presented in the Trace Viewer (ex. Histogram, Flame Chart) are Trace Compass "Views" not Trace Compass "Analyses". (Thank you Matthew for explaining this to me!)

@MatthewKhouzam Could I get a list of most Trace Compass Views?

Letting the user select Views (ignoring the existence of Analyses) feels like the the right first step, so I'm just going make the Trace Viewer terminology consistent with Trace Compass. Examples:

• Change Trace Viewer UI: * "Available analyses" --> "Available views". #292
• Change "analyses" to "views" in this issue name #157 and related issues #156 #158).

Trace Viewer "Analyses"	TC Analyses and Views

Trace Compass Analyses and Views

My understanding is that the existence of "TC Analyses" and "TC Views" comes from the implementation: The trace data transformation path is: Trace events --> TC Analyses --> TC Views

• Analyses: Organization or transformation of data useful for trace-based debugging. Prepared for ease of use in visualisations (or even as input to other analyses?). Often data is from trace files, but it could also be from elsewhere (ex. variable names from the program's source code).
• Views: Visualization of data from one or more analyses.

Are "TC Analyses" an important concept for a Trace Viewer user?

It doesn't seem like it. Contributors/Developers need to know whether they're writing an Analysis or a View, but the user is only interacting with Views, right?

[ebugden]

Listing as many views I can find here to help with creating meaningful view categories.

Category ideas

Categories need to cover all analysis/views and ideally be the same type of thing.

• Physical structure: CPU, Memory, Network, Disk
  • "Computation" instead of CPU (to include GPU, Programmable co-processors)
  • "Storage" instead of "Disk"
  • "Other hardware" to include FPGAs
• Logical structure: Futex/Lock, Scheduler
• Logical action: Context switch
• Logical issues: Contention, Latency

What are the keywords/language that people recognize? What are terms people look for?

Views

Sources: Trace Compass, Trace Viewer, Trace Compass User Guide - Main Features

Data views

What data does each view contain? Why is it useful?

• Context CallStacks
  • Flame Graph: Aggregate view of the function calls from the Flame Chart View, requires userspace data
    • Shouldn't it not have an x-axis if it's a Flame Graph? It's supposed to be aggregated over the entire execution
    • Technically it's an icicle (flame upside down)
  • Flame Chart: Call stack per thread over time, requires userspace data
    • Which traces and what events required to produce this analysis?
  • Stack Table
  • Weighted Tree
• Context Switch
• CPU Usage: %, per process, requires lttng kernel tracer with sched_switch event enabled
  • Confusing y-axis. What does more than 100% CPU mean? One logical CPU is 100% max (hyperthreading) Y value is sum of usage for all CPUs.
    • Motivation from MatthewKhouzam: By keeping CPU usage of 1 core being 100%, we can show that n units are being calculated per second.
  • Where is the legend?
  • Is it per thread or per process?
• Critical Path Aggregated
  • Flame Graph
  • Weighted Tree
• Function Duration: Still exists? requires userspace data
  • Function Duration Statistics
  • Function Duration Density
• Futex Contention: Measures the futexes contention latency between futex entry and exit event for a thread
  • Shows how long a thread stays blocked waiting on a futex?
  • Contention Statistics
  • Contention Table
  • Contention vs. Count
  • Contention vs. Time
  • Scenarios
  • Uaddr vs. Waiter
• Input/Output
  • Disk I/O Activity: Read and write speeds for different disks on the system. XY chart
    • Chart legend is hiding on the far right of the process info
  • Disk Requests
  • Exec Queue Statistics
• IO
  • File Access
  • I/O per process
• IRQ Analysis: Is this latency analysis for IRQ handlers? "measures the latency between the IRQ handlers entry and exit events"
  • IRQ Statistics
  • IRQ Table
  • IRQ vs. Count
  • IRQ vs. Time
• Kernel Memory Usage
  • Y-axis is "relative kernel memory usage" (labelled "memory variation"). Calculated through tracking memory allocation during the program? Meaning we don't know the initial total memory usage (so can't know absolute memory usage)?
• Linux Kernel
  • Control Flow: Thread states "Per-process events." Per process or per thread? Events or time spans? Looks like it shows thread details for the whole trace (when it's running, where it's running, what it's running?)
  • Resources: "State of system resources i.e. if changes occurred during the trace either on CPUs, IRQs or soft IRQs". Interrupts are resources? Shows what CPUs mentioned in the trace are doing (+ CPU frequency)
• Memory Usage: Active memory usage per thread over time, requires userspace data
  • Absolute memory use?
• OS Execution
  • Critical Flow
• Sched Wakeup/Switch Latency
  • Sched Wakeup/Switch Density
  • Sched Wakeup/Switch Latencies
  • Sched Wakeup/Switch Latency Statistics
  • Sched Wakeup/Switch Latency vs. Time
• State Machine / State System Explorer: View internal values of a state system
• Statistics: Counts of how many times a particular event happened
• System Call Latency: Latency between system call entry and exit, per type of system call
  • Latency of a system call or Duration of system call?
  • Per type of system call or per system call? (As in, per function name)
  • System Call Density
  • System Call Latencies
  • System Call Latency Statistics
  • System Call Latency vs. Time

Other data views:

• Histogram: Distribution of Trace events in time
• Event Matching Latency - Latency vs Time
• LTTng-UST CallStack - Latency vs Time
• LTTng-UST CallStack - Latency vs Time
• Ust Memory - Latency vs Time
• Events Table: Raw events list
• Time Chart: Visualize all open traces on a common timeline. Is this a view or a visualisation type used in some of the views ex. Control Flow?

Operation/interaction views

• Colors: Highlight events matching a filter a certain colour
• Filters: Define filter presets. Can be applied to any events table

Misc. Questions

• How many views can only be used with lttng traces?

• Both Context CallStack and Critical Path Aggregated have a Flame Graph View. Does this mean:

  1. Flame Graph can be used to visualize CallStack OR Critical Path Analysis results?
  2. Flame Graph requires CallStack AND Critical Path Analysis results?

• In the implementation architecture, what is the distinction between Analysis and View (Where does Analysis end and View begin)? Does View code only organize and present data (no transformation/calculations)?

---

[ebugden]

@ssmagula @MatthewKhouzam What do you think about the organization below? I'm unconvinced. Here's a rough clustering of some of the Trace Compass views:

• CPU
  • CPU Usage
• Interrupts (IRQ)
• Memory
  • Kernel memory usage
  • Memory usage
• Disk
  • Disk I/O Activity
• Execution timeline
  • Control flow
  • Resources
  • Critical path
• Functions
  • Call stack
  • Function duration
  • System call latency
• Trace events
  • Histogram
  • Table
  • Statistics
• Locking (or Lock?)
  • Futex
• Scheduling (or Scheduler?)

The division is extra confusing because some categories are nouns (physical or logical structures, ex. CPU, Functions) and some are actions (ex. Locking). I considered switching the actions to "Lock" and "Scheduler", but somehow I found those categories more confusing.

---

I think we should leave Views as a flat ordered list for now because:

• I'm struggling to sort all the views into a small number of intuitive categories (after a doc/TC overview to vaguely understand most of the views)
• Right now the Trace Viewer has a small number of functional views. It's not a lot of effort to search in the short list.
• Cleaning up the current views will be higher impact (ex. View names, descriptions), see below.

---

Off topic View-related thoughts after overview:

• Confusing names: Giving the views representative names and descriptions is a high priority for improving clarity #286
• Missing view data requirements: The data requirements for the views are really unclear (what trace types are required? kernel/userspace? what events need to be enabled?)
• Missing graph information: What is being visualized is often unclear (axis titles, legends difficult to discover)
• Sometimes I got the sense that the content of views was dictated more by the implementation (or backend data structure) rather than the analytical utility (ex. This information is stored together so it should be visualized together). This observation could eventually lead to suggestions of changes to views (ex. splitting into two (higher granularity) or different views altogether (different data combinations))
• Some of the views are the same data visualized/organized differently. There could eventually be a more compact way of presenting these (ex. It's one view, but you can choose the visualisation and have multiples open)

View specific:

• Control Flow View: Show active threads by default?
• Critical Path: Order threads based on time so arrows cascade downwards (not scrolling up and down)?
• Histogram: Would feel more valuable if it distinguished between event types (ex. top 3 most frequent events, then other)?

[ebugden]

Not currently relevant since the number of views is short (<10)