Discover help with getting started, esp. compatible file formats

[ebugden]

New users need more help with getting started. Because they don't have a lot invested in the plugin, they need to have smooth onboarding, or they'll give up.

• Users could struggle with: Getting a trace, finding compatible file formats #324, server configuration, getting certain views to work
• Quick fix: Link to GettingStarted.md or Readme.md from Open Trace dialog (and a link to open the Quick Help panel in other versions)

Content

• What is tracing? (ex. Point to the information in the first couple trace labs)
• Point to how to install the trace viewer locally? (Or test if can upload typically sized first traces to Gitpod. It's possible to upload folders to the browser application (drag & drop), but I don't know if the tracevizlab traces are fairly small for traces.)
• Simple way to collect a short but interesting trace (ex. Using ftrace)
• Which views to look at, what to look for

Quick help should point to:

• External tracer doc when possible to avoid duplicating information that could eventually change (ex. lttng documentation, ftrace kernel readme)
• Useful helper scripts (ex. To adjust lttng configuration, record trace helper script)

Implementation

⚠️ Keep in mind @bhufmann comment:

• The getting started guide depends on the supported trace types in the trace server.
• The theia-trace-extension front-end is server-agnostic, for example, there might be trace servers that don't support LTTng or fTrace.

Brainstorming

bhufmann: I think the trace server needs to provide the detailed information about supported trace types, getting started help etc. That would require a trace server protocol endpoint that the front-end can query and then display to the user.

• Having the server send info about supported trace types and getting started help seems like the most well integrated solution, but it also involves changes to the server, tsp and front end which seems like a lot of effort.
• My first intuition was to initially implement a solution that is less integrated, but easier to implement to allow quick user testing and less complex iteration (ex. Make the front end help link easy to configure (so it can point to a file, external link, or help widget)). But, design iteration could be done with a wireframe instead.

Having the trace server provide:

• Information about supported trace formats and types: I agree. This makes complete sense based on the server's responsibilities.
  • Can the server easily generate a list based on available analyses?
• Getting started help: It makes sense for the server to provide documentation or to configure help links (based on supported traces).
  • More complicated to adjust?: Would adjusting the help file that is sent require recompiling the server? After a change to the list of supported trace types, recompilation makes sense, but for changing the help link that is sent recompilation seems like overkill?
  • Integrated help (ex. Coach marks): The server wouldn't be able to provide more integrated forms of help like coach marks. These would be implemented in the front end, but the server could still be responsible for triggering them (based on supported traces).

Some less integrated (but easier to implement?) solutions:

• Hardcoded link to Linux-specific help documentation?
• Make the front end help link configurable?
• Have the server specify supported trace types, then configure the help link based on that? (As in, only show ftrace help if ftrace traces are supported) (changes server, tsp, front end)

---

Issue source: @ssmagula's user tests + brainstorming with TC and tool team members

[ebugden]

The Linux RT Wiki has some pages it could be fun to point beginners to:

• RT tools and utilities (ex. Ftrace)
• Steps for latency debugging

Disclaimer: These were written by pre-information-design Erica so they sound pretentious and are heavy to read

[ebugden]

One suggestion is to create a "Getting started with tracing" help page (maybe on the Wiki here or a section in the project README) that helps new users get traces if they don't have any. A rough draft for this page is below.

---

Installing the tool locally

This allows you to view your own traces. See how to capture your own trace data below

• Download the Trace Viewer application binary
• Download & configure trace server

To try the tool without tracing yourself, you can download example traces from the Trace Visualization Labs. To avoid installation and setup entirely, take a look at the live demo.

Capture a Quick Kernel Trace

Here's one way you can capture your own trace data to visualize!

• Operating system: Linux
• Tracer: LTTng

Estimated time: 15 min

(Other Linux kernel tracers: Ftrace (point to instructions? ex. Real time wiki instructions))

1. Install LTTng tracer for kernel tracing

The LTTng installation documentation has distribution-specific instructions for how to install.

On Ubuntu, the commands would look something like:

# apt-get install lttng-tools (Libraries and command-line interface to control tracing) # apt-get install lttng-modules-dkms (Linux kernel modules to instrument and trace the kernel)

• How to check that it's installed correctly? Is there a lttng --version?

2. Trace the Linux kernel using LTTng

The LTTng kernel tracing quick start shows how to enable trace points in the kernel and then collect trace data.

• Which kernel events should we suggest they enable to get a good number of analyses?
• What operation should we suggest they do while tracing to get a reasonably interesting trace? Run your program
• Where should we suggest they put the traces (instead of /tmp)
• Point users to a script to set good defaults?

All together, the commands would look something like: # lttng create kernel-session --output=~/quick-kernel-trace # lttng enable-event --kernel sched_switch,sched_process_fork # lttng enable-event --kernel --syscall open,close # lttng start [do something exciting like loading a website] # lttng destroy

3. Keep your kernel trace in a permanent place

• Make sure they're not in /tmp (disappear when log out)
• Do we want to make ~/traces a thing?

Now you can take a look at your kernel trace with the Trace Viewer! Alternatively, kernelshark is a well known kernel trace viewer.

Dev setup

If you want to make modifications to the code or make contributions quick changes: gitpod download code

[ebugden]

• Display documentation in directly in the tool? Open a tab instead of opening the browser (makes the documentation feel more integrated in the tool and trustworthy)

• Package quick start documentation with the tool?

Feedback from @arfio

• First tracing experience should be something higher level, userspace (ex. strace, lttng-ust). Maybe introduce them to gantt charts
• Point users towards a couple basic trace viz labs (for capturing their first traces). (ex. LTTng userspace tracing, [Kernel trace with Ftrace] (https://github.com/tuxology/tracevizlab/tree/master/labs/004-record-kernel-trace-ftrace))
  • Add a trace viz lab that is a super simple trace and super simple analysis
  • Completing a couple labs is sufficient for users who are casually exploring the tool, but want to try capturing their own traces
  • The userspace tracing lab is really good for understanding a program, but it shouldn't be a user's first trace
• Might be better to point users towards ftrace, perf initially
• Point exprienced users towards lttng. Tracing with lttng is not the best example for tracing beginners
  • Lttng has frequent and difficult to resolve installation problems (ex. modules won't build, modules won't load). These are difficult to resolve even for advanced users.
  • Lttng requires special privileges: Member of tracing group, or sudo for every operation
  • Kernel traces are difficult to decipher for beginners
  • Point to Ubuntu installation instructions in trace viz labs?
  • Point to record trace helper script?
  • To test that lttng is installed correctly: lttng list -k
• If users are exploring the tool because they have a specific problem they want to solve, they are willing to dedicate more time (ex. 1h) to explore, but it's important to have examples (ex. viz labs) that are relevant to their domain / use case.
• Look at other analysis tools (ex. more commercial ones) to see how they onboard users:
  • Intel VTune
  • Vampir
  • ARM MAP

---

Feedback from @ssmagula

• Would userspace tracing be a better first example for beginners?
• Would ftrace or perf traces be a better first trace? Are they compatible with enough views? MatthewKhouzam: For FTrace and perf, the threads/Resources/CPU Usage views work. In Trace Compass classic there are a lot more.

[bhufmann]

@ebugden the getting started guide depends on the supported trace types in the trace server. The theia-trace-extension front-end is server-agnostic, for example, there might be trace servers that don't support LTTng or fTrace. The domain specific logic and data is in the trace servers. The getting started guide can't be entirely implemented on the front-end side only. I think the trace server needs to provide the detailed information about supported trace types, getting started help etc. That would require a trace server protocol endpoint that the front-end can query and then display to the user. What do you think?

[ebugden]

bhufmann: the getting started guide depends on the supported trace types in the trace server. The theia-trace-extension front-end is server-agnostic, for example, there might be trace servers that don't support LTTng or fTrace.

True! I'll add this important context to the issue description. Thank you.

bhufmann: I think the trace server needs to provide the detailed information about supported trace types, getting started help etc. That would require a trace server protocol endpoint that the front-end can query and then display to the user. What do you think?

For now I'm focusing on defining the content for a reasonably widely applicable help document (tracing on linux with ftrace) that could be tested and iterated on using a wireframe. I'm going to think more seriously about implementation after that.

I added more detailed thoughts in the implementation section of this issue description.

[ebugden]

WIP: 2nd draft for "Getting started with tracing" help page.

Changes:

• Focus on higher-level (ideally application-level (userspace) rather than system-level (kernel)) examples
  • Docker container with lttng-userspace installed? Would that be easier than trying to get people to understand kernel traces right away?
• Focus on tracers that are included in the kernel: ftrace, perf (don't need to go through a possibly complex installation process).
  • The existing ftrace and perf labs make many mentions of lttng and Trace Compass which would be confusing for new users (even if we say: ignore all these distracting/confusing terms). It could be appropriate to write a slightly edited version of the lab for the quickstart doc. Or rewrite the labs completely but be inspired by them.

Since the UI is moving around so much at this stage, maybe just point towards what tracing is, why it's useful, and how to get some traces. Maybe explain CTF folder structure. They'll have to manage on their own for opening traces and navigation.

Record kernel trace with ftrace Record a kernel trace with perf RT Wiki Ftrace doc

Which terms below will beginners not understand (ex. kernel)?

---

What is tracing?

What is tracing? Why use tracing? When to trace?

Use cases: Crash, Performance optimization, Structural code analysis

Capture a quick kernel trace

Here's one way you can capture your own trace data to visualize!

• Operating system: Linux
• Tracer: ftrace (available by default in most Linux distributions)

Setup ftrace

Remove this section? Just have them try to mount it in a script and have external resources they can go explore if something goes wrong. I think that's better. They don't necessarily need to understand everything at first. We just want to get them to a visualisation as quickly as possible.

• Mount tracefs

Record a kernel trace

Record kernel trace with ftrace

TODO: Change so that mounting and enabling events via tracefs not debugfs

# Mount debugfs (TODO: change script so mounting tracefs)
sudo mount -t debugfs nodev /sys/kernel/debug # Remove this line? Move to previous section?

# Enable tracepoints (events) (TODO: Enable events via tracefs)
sudo echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/sched/sched_waking/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/sched/sched_pi_setprio/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/sched/sched_process_fork/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/sched/sched_process_exit/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/sched/sched_process_free/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/irq/softirq_raise/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/irq/softirq_entry/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/irq/softirq_exit/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/irq/irq_handler_entry/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/irq/irq_handler_exit/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/block/block_rq_complete/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/block/block_rq_insert/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/block/block_rq_issue/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/block/block_bio_frontmerge/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/power/cpu_frequency/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/net/net_dev_queue/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/net/netif_receive_skb/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/timer/hrtimer_start/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/timer/hrtimer_cancel/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/timer/hrtimer_expire_entry/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/timer/hrtimer_expire_exit/enable
sudo echo 1 > /sys/kernel/debug/tracing/events/syscalls/enable

# Start tracing
sudo echo 1 > /sys/kernel/debug/tracing/tracing_on

# Something to trace
wget https://lttng.org

# Stop tracing
sudo echo 0 > /sys/kernel/debug/tracing/tracing_on

TODO: Add downloadable script?

Analyze the trace

[ebugden]

This issue came up again during a user-centered planning process.

Need: Help me learn what and how trace

As a new user, I need some guidance and assistance to get started with trace recording, so that I can go from being a total newb, to a novice more quickly and with less pain.

Provide:

• Tutorial text?
• Bash script(s)?