Extensible eBPF timeline attributes

[wks]

The purpose of this PR includes:

1. Providing a systematic way to add additional attributes to events on the timeline generated from eBPF traces.
2. Adding additional USDT probes in mmtk-core and display more useful information on the timeline.
3. Allowing users, especially VM binding developers, to extend the tracing scripts to visualize additional USDT probes in the binding.

In previous PRs https://github.com/mmtk/mmtk-core/pull/1154 and https://github.com/mmtk/mmtk-core/pull/1158, we use "meta" events to add attributes to work packets, including the number of slots processed by ProcessEdgesWork and the number of allocated blocks visited by SweepChunk. This PR refactors the scripts to standardize the use of "meta" events. Specifically

• The Rust program executes USDT trace points to expose additional details (such as the number of slots processed) during the execution of a work packet
• capture.bt emits "meta" events for such USDTs, and
• visualize.py adds attributes to the current unfinished GC event and/or work packet event of the current thread.

In this way, the information about a GC or a work packet can be gradually added by multiple USDT trace points, providing more flexibility than having to provide all information at the beginning or the end of the work packet.

Using this mechanism, we added a few more USDT trace points in mmtk-core, including:

• Generation plans trigger a USDT trace point when they have decided if the current GC is a full-heap GC. The information is displayed on the big "GC" bar (in "Thread 0") for each GC on the timeline.
• Immix-based plans trigger a USDT trace point when they have decided if the current GC is a defrag GC. Displayed on the "GC" bar, too.
• Root-scanning packets now have "roots" attributes showing how many root slots or root nodes are reported.
• ScanObjects work packets (as well as ProcessEdgesWork packets that scan objects immediately) now display the number of objects scanned.

We also introduce extension mechanisms to capture.py and visualize.py. Both of them now accept the -x or --extra command line argument. Specifically,

• capture.py uses -x to specify an additional script to be appended after the capture.bt script.
• visualize.py uses -x to specify a Python script to handle unknown event names.

The extension mechanism allows VM binding developers to insert custom USDT trace points in the VM binding, and extend the timeline tools to visualize those custom events.

[wks]

I haven't gone through the entire PR yet, but the overall direction looks good.

I think it would be useful to open an mmtk-openjdk PR as well. I know this PR provides some examples, but the real extension .bt/.py scripts should be shipped and versioned together with the binding, because it depends on the internal implementation of the binding.

I opened a PR for mmtk-openjdk. It will use the extensible eBPF tools introduced here. But since its main purpose is making the code cache roots generational, it should be merged separately after this PR.

The extensible eBPF tool is most useful for root scanning and weak reference processing. But the status quo of the OpenJDK binding is,

1. For root scanning, we simply delegate each kind of root to a C++ function. This PR already adds USDT tracepoints for root scanning, counting the number of root slots or root nodes. That's already enough for most roots. We may refactor the code cache roots (in https://github.com/mmtk/mmtk-openjdk/pull/282) and probably the ClassLoaderDataGraphRoots, too, and add more trace points to see more details.
2. We currently disable weak reference processing by default. Even when enabled, we offload it to the ReferenceProcessor and FinalizerProcessor in mmtk-core. We may add more tracepoints in the reference/finalizer processors in mmtk-core, but I think it is more interesting if we can observe the details of weak references and weak tables (such as the interned string table) on the VM binding side.

Meanwhile, I am working on the Ruby binding (https://github.com/mmtk/mmtk-ruby/pull/81) and it will make use of this feature, too.

My plan is merging this PR first and then work on each VM binding to make use of this feature.

[caizixian]

This PR makes extensive use of |= syntax on dictionaries. This would require PEP 584 introduced in Python 3.9 https://docs.python.org/3/whatsnew/3.9.html

Ubuntu 20.04 is still a supported LTS and uses Python 3.8 by default. I wonder whether we should err on the safe side and use .update instead (we just need in-place update in this case). Same for the code snippets in EXTENSION.md.

Similarly, we have some places that use match-case, which would require Python 3.10, it shouldn't be too bad to rewrite it using plain if-elif-else.

I understand that Ubuntu 22.04 already has Python 3.10, so I don't have a strong opinion on this. But I do think it would be better to be more conservative so that MMTk users run older distributions won't get unexpected syntax errors. If anything, we should mention the Python version requirement in the README.

[caizixian]

We should probably also add some more comments to the visualization script (in particular docstrings for enrich_meta and enrich_events for the arguments).

[caizixian]

The PR otherwise looks good.

[wks]

This PR makes extensive use of |= syntax on dictionaries. This would require PEP 584 introduced in Python 3.9 https://docs.python.org/3/whatsnew/3.9.html

Ubuntu 20.04 is still a supported LTS and uses Python 3.8 by default. I wonder whether we should err on the safe side and use .update instead (we just need in-place update in this case). Same for the code snippets in EXTENSION.md.

Similarly, we have some places that use match-case, which would require Python 3.10, it shouldn't be too bad to rewrite it using plain if-elif-else.

Personally I am not very opinionated about |= or .update. I think both are readable enough. But I strongly advocate for using the match statement because if ... elif ... is simply too verbose and unpleasant to use. I have been blaming Python for not having a match statement since the Python 2.5 era.

I understand that Ubuntu 22.04 already has Python 3.10, so I don't have a strong opinion on this. But I do think it would be better to be more conservative so that MMTk users run older distributions won't get unexpected syntax errors. If anything, we should mention the Python version requirement in the README.

I think it is OK to require a more recent Python version rather than having our hands tied. bpftrace requires sudo privilege to run, so the user should have enough privilege to install a more recent version of Python 3.10 if they have to run Ubuntu 20.04. I'll make a note in the README file, and double check if it actually runs with Python 3.10.

[wks]

We should probably also add some more comments to the visualization script (in particular docstrings for enrich_meta and enrich_events for the arguments).

I added some docstrings for methods which extension developers may be interested in. That includes enrich_meta and enrich_events and their arguments.

[caizixian]

I think it is OK to require a more recent Python version rather than having our hands tied. bpftrace requires sudo privilege to run, so the user should have enough privilege to install a more recent version of Python 3.10 if they have to run Ubuntu 20.04. I'll make a note in the README file, and double check if it actually runs with Python 3.10.

Yeah, I think it's a reasonable tradeoff.