Discover compatible trace files

[ebugden]

Finding compatible trace files is required for using the Trace Viewer. During user tests, an experienced dev and a less experienced dev were both unable to complete these tasks without significant coaching. Users must be able to comfortably:

• Locate trace files - Where are trace files?
• Identify compatible trace files - Which files are trace file? Which trace files are compatible formats?
  • Nudge users towards CTF subtypes that support the most Views
• Identify what to select to correctly open the trace - Ex. What is the root of the trace file?

Challenges

• Locate trace files
  • Default trace location varies between tracers
  • Users can often change where traces are generated
• Identify compatible trace files
  • Filenames, file endings are not fixed (ex. CTF files don't necessarily end with .ctf)
  • Different trace types have different structures (ex. folder vs. file)
  • Large number of compatible trace types

Proposed solution elements

• Magic Folder: A user-configurable folder. User identifies folder paths that should be watched by the Trace Viewer and scanned for traces. Detected files automatically appear in UI.
• Scan system for compatible trace files #8
  • Should file hierarchy be kept or not? Ex. Fade out non-compatible files in the browsable file structure? Only show a list of detected compatible files (squash all file hierarchy)?
• Explicitly filter for certain trace formats: Open Traces dialog presents a user editable filter, initially set to 'CTF'? NB: User could always dismiss the filter, and browse to any and all files and a tree filter already implemented in Theia. Default filters could be configurable by settings.

Related: #349

---

Issue source: @ssmagula's user tests + brainstorming with TC and tool team members

[bhufmann]

Discover of trace files are domain and tracer specific, where the tracer is the application that produces traces.

First of all the location where traces are going to be stored is not fixed across tracers. For example LTTng produces their traces by default into the user's home directory under ${user.home}/lttng-traces. The location can be configured by users.

Secondly, the file format, filenames, file endings are not fix either. LTTng, for example, use the CTF format which is a directory with a metadata file and binary files. Other tracer may create traces in JSON format some text format. Just looking on the file extension won't be sufficient.

Discover of compatible trace files need to be configurable and extensible.

[ebugden]

Thank you for the details! I updated the description with these challenges.

[ebugden]

What would our version of the "popular trace formats" list be?

Ex. Perfetto claims it's interoperable because it "can import and export popular trace formats: Chromium JSON trace format, Android Systrace, ftrace, CSV"

@bhufmann @MatthewKhouzam @arfio What are the 3-5 most common trace file formats:

• For the whole tracing universe?
• For our intended users?

---

bhufmann: file format, filenames, file endings are not fix either. LTTng, for example, use the CTF format which is a directory with a metadata file and binary files.

Right now, Trace Compass is able to identify CTF files without making any assumptions about filenames, right? I want to confirm that the code to detect CTF files exists already.

[MatthewKhouzam]

In my unscientific opinion:

• Syslog (/var/log)
• OTW2 (Intel VTune)
• CANBus (hardware in cars)
• JTAG (hardware traces)
• PCAP (Wireshark (doo-doo-doo-doo-doo))
• windows etw (98% of the desktop market)
• maybe json (Chrome and opentelemetry both use Json with different internal formats)

For our intended users:

At the moment, we are catering to the LTTng crowd. So CTF. Also, they have some JSON chromium style user space traces. Chrome tracing has a huge advantage over CTF that when you write the trace, the output is known. (i.e. there's only one way to write counters etc...)

I strongly believe we should work towards coupling LTTng's precision with open tracing's wide scalability so we can drill down on transactions. Like Loic G's masters brought forwards.

About intended audience. We really have three: students, sysadmins and devs. They all have different requirements. Bear (:bear: ) in mind, these are vast generalizations. Students use this to visualize basic OS stuff often. Admins look at logs in a post-mortem fashion Devs will make several trace runs and compare them.

[bhufmann]

Right now, Trace Compass is able to identify CTF files without making any assumptions about filenames, right? I want to confirm that the code to detect CTF files exists already.

With context-sensitive menu "Open With -> Trace Viewer":

• if user selects a directory then the open procedure looks for all CTF traces recursively, by checking if a file with name "metadata" is present, and then puts it in one experiment.
• If the user selects a single file, then this file is uploaded to the server, if the server is able to open it as trace then it's considered as trace, otherwise the server will reject that file and the user is notified.

With the "Open Trace" dialiog, only directories can be selected and hence the scanning for CTF is done as above.

Just to be clear, code to detect CTF is based on that a file with name metadata is present in a directory. The server will reject a directory that contains a file metadata which is not a CTF metadata file.

[bhufmann]

About the magic folder mention earlier, I think the magic folder is am interesting idea to restrict the location where the tool is looking for traces. Scanning the file system might be costly.

[ebugden]

Just to be clear, code to detect CTF is based on that a file with name metadata is present in a directory. The server will reject a directory that contains a file metadata which is not a CTF metadata file.

Ok so the code takes a good guess that it's a CTF file, but right now the trace server has to attempt to open the trace to know for sure it's a CTF file.

[ebugden]

@MatthewKhouzam or @bhufmann Could you list and explain a bit about the different CTF subtypes and also say which allow the most Views? This will help us determine which filters would be most useful and which types we should nudge users towards.

Could you also explain the difference between trace format and trace type?

[ebugden]

Trace format: A technical specification of how to present the trace data. Examples of formats: Opentrace, Trace Event Format (JSON traces generated by Chrome), Common Trace Format (CTF). Can be composed of one file or multiple files in a folder.

• Customized CTF format: CTF is highly customizable. All Trace Compass analyses are not compatible with all forms of CTF traces. Especially for application-level traces, there tend to be lots of customizations (so custom analyses must also be written).

---

Trace type (internal to Trace Compass): An internal label that defines which analyses are compatible with the trace. Trace type is detected automatically (when a trace is opened at the root as the tool expects).

• Could be useful for categorizing traces
• Which internal trace types have the most views?

[ebugden]

@hriday-panchasara @mirsky-work If we want to quickly make this slightly easier for users in time for the MVP, @ssmagula and I suggest adding a message at the top of the "Open Trace" dialogue pop-up, ex. "Having trouble finding trace? Learn more about typical trace locations." that has a configurable link to documentation.

The message and link would be in this general area:

[mirsky-work]

@hriday-panchasara @mirsky-work If we want to quickly make this slightly easier for users in time for the MVP, @ssmagula and I suggest adding a message at the top of the "Open Trace" dialogue pop-up, ex. "Having trouble finding trace? Learn more about typical trace locations." that has a configurable link to documentation.

Thanks for the suggestion! I agree it would have been nice, but we are using Theia's FileDialogService for the Open Trace dialog and it doesn't seem it supports such a customization. Maybe we can implement a similar dialog ourselves, but it won't be straightforward. I'd prefer to examine other options.