microsoft / microsoft-performance-toolkit-sdk

Software Development Kit for the Microsoft Performance ToolKit
MIT License
156 stars 55 forks source link

Add support for specifying variants of added columns during table building #370

Closed mslukebo closed 1 month ago

mslukebo commented 1 month ago

Overview

This PR exposes three APIs that together address #208 and #237, and closes #373:

  1. Add APIs in the runtime for plugins to specify column modes during table building.
  2. Add APIs in the runtime for SDK drivers to discover and fetch modes registered for individual columns.
  3. Add APIs in the Engine for fetching registered modes

A column mode is defined as an alternative IProjection<int, T> for a column. When adding column modes during table building, plugins supply

  1. A ColumnVariantIdentifier that contains a Guid uniquely identifying the added variant
  2. An IProjection<int, T> to use for that variant. Note that variant projections need not return the same type as the base column's type.

Registered column variants are exposed to consumers as an untyped IDataColumn whose ColumnConfiguration is the same as the configuration of the variant's base column.

Column variants also have meta-information about how the variants relate to the base column and each other. For example, a base column of type IDataColumn<Timestamp> may have a variant projection IDataColumn<DateTime> that exposes the same time value as a DateTime time relative to some fixed point in time. The meta-information is that the base column can be toggled to "project timestamp as DateTime".

Continuing with this example, suppose the base Timestamp column can project to different DateTimes that are relative to different points in time (e.g. one DateTime relative to the start of the trace and one DateTime relative to the first event). In this scenario, the meta-information is that the base column can be toggled to "project timestamp as DateTime" in one of two modes.

This meta-information is defined during table building through the API exposed to plugins. Variants are registered through ColumnBuilder classes, where both the methods used and the order that they are used dictate variants' relationships. For concrete examples of how this looks, refer to ColumnVariantsTests.cs

Meta-information along with concrete column variants are exposed as a tree structure accessible through a new IColumnVariantsRegistrar interface. This interface is exposed on the runtime's TableBuilder class (but not the ITableBuilder interface passed to plugins). The IColumnVariantsRegistrar allows SDK drivers to fetch IColumnVariantsTreeNode instances that represent the root of registered variants. Drivers can implement a IColumnVariantsTreeNodesVisitor that can traverse the tree data structure to understand both

  1. The meta-information defined during table building
  2. The concrete variants associated with different nodes in the tree, exposed as IDataColumns on tree nodes

Finally, the Engine's ITableResult was updated to expose all possible ColumnVariantIdentifiers for IDataColumns on the table. This representation loses the meta-information that defines how the variants relate to each other, which is OK since the Engine is used for programmatic access to data where the conceptual relationship between variants is not necessary.

Column Builders API Design

To be consist with the "builder" architecture used to construct tables, plugins use a builder pattern to add variants of a column. The following methods were added:

  1. ITableBuilderWithRowCount.AddColumnWithVariants
  2. TableBuilderExtensions.AddColumnWithVariants
  3. TableBuilderExtensions.AddHierarchicalColumnWithVariants

These methods follow the same pattern of AddColumn and AddHierarchicalColumn, where the ITableBuilderWithRowCount method takes in the IDataColumn being added and the TableBuilderExtensions methods take in the column configuration + projection. These three methods have an additional parameter Action<IRootColumnBuilder> variantsBuilder that allow plugins to provide a callback that configures the added column's variants. A plugin's code might look like

tableBuilder
    .SetRowCount(1)
    .AddColumnWithVariants(baseConfig, baseProj, builder =>
    {
        builder
            .WithToggle(projectAsDateTime, utcProj)
            .Build();
    });

There are two differences between the column builders' design and the table builders' design:

  1. Column builders are purely functional with no mutable state. Every method on the builder returns an entirely new object
  2. Plugins must call Build() on the column builder for any change to have an effect

This decision was made because, unlike table builders, both the final state and valid operations of a column builder depends on the order methods were called. For example, once a base column has modes added, it should not be able to add a toggle at the same level of the group of modes (adding toggles beneath an individual mode, however, is valid). I.e., this must be invalid:

tableBuilder
    .SetRowCount(1)
    .AddColumnWithVariants(baseConfig, utcProj, builder =>
    {
        builder
            .WithModes(utc.Name)
            .WithMode(local, localProj)
            .WithToggle(projectAsDateTime, utcProj) // INVALID CALL!
            .Build();
    });

Ensuring WithToggle cannot be called after WithModes is added is accomplished by the return type of WithModes being a different interface than IRootColumnBuilder. Specifically, WithModes returns an IModalColumnBuilder that does not expose a WithToggle method.

However, this is not enough to ensure invalid states are impossible. If the column builder was a single, stateful object, the plugin could write this code:

tableBuilder
    .SetRowCount(1)
    .AddColumnWithVariants(baseConfig, utcProj, builder =>
    {
        builder
            .WithModes(utc.Name)
            .WithMode(local, localProj);

        builder
            .WithToggle(projectAsDateTime, utcProj) // INVALID CALL!
            .Build();
    });

If builder was not a functional object, it would be forced to throw an exception on the invalid call. Instead, the behavior of the above code is to completely disregard the added modes since the object that was actually built was just a single toggle. The built column will therefore have just a single toggle underneath the base column.

Column Builders Implementation

There are several classes that implement the various ColumnBuilder abstract classes depending on what has been added. For example, there is one implementation of an RootColumnBuilder that has had nothing added to it.

This PR adds an IColumnVariantsProcessor interface to the runtime. This interface is passed to ColumnBuilder implementations to be invoked during their Build calls. When a builder is built, it constructs a column variants tree based on that builder's data and then passes it to the processor it was given.

The runtime also defines a helper interface IBuilderCallbackInvoker that lets column builders fetch an IColumnVariantsTreeNode for children of tree nodes it is building. The implementations of this interface run the plugin-supplied Action<T> where T : ColumnBuilder callbacks, returning the final root tree node built by the callback.