CDash: Add upload, storage, display, and download of additional enhanced build and test metrics

[bartlettroscoe]

Feature Request

Related issues

• cmake#26247: Add collection and reporting of additional build and test metrics to CMake, CTest and CDash
• snl_kitware#265: EPIC: CMake, CTest, CDash: Add collection and reporting of additional build and test metrics to CMake, CTest and CDash (protected project)

Description

This is the companion CDash issue for the CMake, CTest, CDash upload request cmake#26247. On the CDash side, we need it to be able to store metrics for every build target and every test, provide some views of that data, and extend the CDash REST API to be able to flexibly download that data.

There is a lot that needs to be specified on both the CMake, CTest and CDash sides to make this work.

ToDo: Flesh out more detailed requirements and use cases.

Requirements for Minimum Viable Product

For the initial driving use case, we need a **Minimum Viable Product (MVP) on the CDash site to go along with the MVP on the CMake/CTest side to have the following features/behaviors:

1. Accept and store enhanced build target-based data submitted by CTest (or the same files CTest would submit for projects that don't use CTest).
2. Accept and store enhanced test data submitted by CTest (or the same files CTest would submit for projects that don't use CTest).
3. Provide some means to download that enhanced build target-based data for a given build on CDash using the CDash REST API (i.e. as specified by Group, Site, BuildName, and BuildStartTime).
4. Provide some means to download that enhanced build target-based data for a given build on CDash using the CDash REST API (i.e. as specified by Group, Site, BuildName, and BuildStartTime). (This could be nothing more than the already supported REST API feature to download build files.)

NOTES:

• For the MVP, target-based build data could be handled as nothing more than a single JSON file submitted and stored as a build file, e.g. using ctest_upload() and that JSON file could be downloaded as a build file. Anything is fine as long as it is relatively efficient in storage and speed (so it can scale).
• For the MVP, we would not need any viewing of the enhanced target-based build data on CDash. We just need the ability to download it for a given build in a form that we can easily parse and analyze (e.g. with a Python script).
• For the MVP, enhanced test data, this extra test data could just be handled as additional test "measurements" that is already supported on CDash. And that would allow their download as well with the existing CDash REST API.

[williamjallen]

Storing the enhanced test data via the existing "test measurements" functionality in CDash could be done with no changes to CDash. The data can be downloaded via the GraphQL API in CDash 3.6+ thanks to https://github.com/Kitware/CDash/pull/2406.

While uploading build data files to CDash could work in theory, doing so would lead to unnecessary backwards compatibility complexity down the road. I think it would be far better to do it "right" by storing build measurement data in dedicated tables in the CDash database. For a basic MVP, this could simply be stored in the database and exposed via the GraphQL API with no UI to start with. This approach would probably take a week or less to implement once the requirements are worked out.

[bartlettroscoe]

@williamjallen, whatever you do, we will be stress testing this pretty quickly by generating build metrics for all of Trilinos. (It is not as many targets as Sierra but it will give us an idea about the performance.)

[williamjallen]

I just opened https://github.com/Kitware/CDash/pull/2501 which adds the ability see and filter the build measurements for a given build.

[bartlettroscoe]

@williamjallen, can we differentiate the type of target:

• object
• library
• executable
• custom (i.e. added with add_custom_command())
• other (everything else)

?

Also, can this also display the labels associated with each target and provide a way to filter based on the labels? (I think we can use CMake target labels to differentiate library targets/code from test targets/code.)

I will add that to the wish-list.

[williamjallen]

@bartlettroscoe Cross-post from https://github.com/Kitware/CDash/pull/2501#issuecomment-2423230610:

Yes, there will be a field which contains information what type of command was run. Currently the placeholder value is "TARGET", but that will change once we agree on the final design of the CTest-CDash interface.