CICD: Output xUnit Test Metrics

[spoonincode]

There is a desire to collect metrics & statistics on the number of regression tests run in the CI workflows. This is not easy to collect currently: a Tests/NP Tests/LR Tests job in a workflow will contain any number of tests (currently 11 through 2000+) and the only way to resolve the exact number is to look at the log output (which I wouldn't consider a stable interface).

It appears that a list of test names and those that failed as well as timing information for each test is the minimum reporting requirement.

I propose we output some sort of machine readable (i.e. stable format) file with test statistics as part of the workflow run that external tooling can more easily access and ingest. ctest's --output-junit seems like it might be a good start as it provides the needed information.

A couple problems that need to be considered:

• Only Ubuntu 22's ctest is new enough for --output-junit. My hunch is it's okay to just collect these statistics on ubuntu22; the tests are the same across distros anyways. But, I was hoping to avoid things like if(ubuntu22) ctest --output-junit; else ctest sort of stuff in the workflows as that quickly ends up spaghetti.
• We will end up with dozens of .xml files, and we really need to somehow coalesce these to a single file. That single file doesn't need to be the original XML format: just something in a stable machine readable format. Could be XML, JSON, or whatever. How and when to do this in the workflow though? Do we need a separate "collect and produce test stats" job? Will there be leftover intermediate XML artifacts between jobs if doing that? Since it's only a list of test names & timing maybe we could post process the XML in each pertinent job and just store the information as an output to be coalesced in a file "collect and produce test stats" job.

[kj4ezj]

We used the -T Test arguments in EOSIO repos to generate xUnit XML artifacts from ctest.

ctest -j "$(nproc)" --output-on-failure -T Test

This left an xUnit XML file at /build/Testing/${dateTime}/Testing.xml. We pulled this file out and parsed it using xml2js in JavaScript. We packed the data we wanted from this into a JSON blob and attached it to the build as test-metrics.json.

I think we can copy/pasta much of that code and follow a similar pattern.

Our implementation should be far simpler. We do not need to interact with the Buildkite API, or to try to diagnose failures, or any of that fanciness. We can leave an xUnit XML file on the disk at the end of each test step, then have a subsequent job step that runs a simple JavaScript program to parse the file directly from the disk, then a third job step can upload it as an artifact. Simple.

[kj4ezj]

Note that xUnit is the language-agnostic name for jUnit test output. This confuses some people. It is the exact same syntax and stuff, just the 'j' stands for "Java" so the language-agnostic term for this is xUnit.

[spoonincode]

Much of what you're suggesting sounds like it should be implemented in actions & workflows that are external to Leap repo. That way such metrics gathering can easily be incorporated into other repos like cdt. Plus, I really want to keep the leap repo's CI as simple as possible (when I start hearing copy pastaing old b1 CI stuff I get really worried), but squirreling away stuff in other actions or workflow repos is a little more palatable.

For example, the current workflow may need only a few lines added to it, such as

      - name: Run Parallel Tests
        run: |
          cd build
          ctest --output-on-failure -j $(nproc) -LE "(nonparallelizable_tests|long_running_tests)" -T test
      - name: Forward XML metrics
        uses: AntelopeIO/forward-xml-metrics-action@v1
        with:
          name: ${{matrix.platform}}-tests

And then later a reusable workflow that does all the bundling and output

  metrics_reporter:
    uses: AntelopeIO/xml-metrics-reporting-workflow/.github/workflows/xml-metrics-reporting-workflow.yaml@v1
    needs: [tests, np-tests]
    with:
      inputs: ['ubuntu20', 'ubuntu22']

There are certainly some challenges to the above such as how to pass around outputs/files and discover the names of the outputs/files (matrix jobs complicate this some), I know I'm over simplifying it some -- some additional lines may need to be required for such grief. But I'm hoping we can avoid adding a bunch of bash and stuff to the main leap repo.

[stephenpdeos]

We will be modifying this issue to capture the github actions and repos associated building building and testing. Remainder of work will be introduced in new issues to follow in future sprints. @kj4ezj to author follow-on issues.

[bhazzard]

This was PMO driven. @BenjaminGormanPMP to determine priority of this issue going forward.