Feature request

Type

[x] - Enhancement - completely new feature
[ ] - Improvement - make what we have better

Is your feature request related to a problem?

We want to collect test run data from local machines (Laptops) to analyse and help improve the developer experience. We should be able to correlate this with data from CI to look for common behaviour that indicates poor local experience.

For example:

People not running test on their local, pushing to CI for testing
Specific test suites that only run on CI that people never run locally
Test that take a long time to run locally when compare to CI, or both CI and local are long
Tests that're repeatedly run on without code change to succeed, indicating flakiness (might be tricky)

Describe the solution you'd like

Most of our projects use nunit, so we can start here.

Nunit has an Engine Extension that we can use to create an installable package.

https://docs.nunit.org/articles/nunit-engine/extensions/creating-extensions/Writing-Engine-Extensions.html

It requires these .addins files to register it (they just need to be a content item in the package tools folder see example here)

https://docs.nunit.org/articles/nunit-engine/extensions/Installing-Extensions.html

Then there is this extension Type we can use to listen to test events.

https://docs.nunit.org/articles/nunit-engine/extensions/creating-extensions/Event-Listeners.html

public interface ITestEventListener
{
    /// <summary>
    /// Handle a progress report or other event.
    /// </summary>
    /// <param name="report">An XML progress report.</param>
    void OnTestEvent(string report);
}

The event takes a XML documents as string input so we'll need to parse the string and it'll be one of these event types

Start of run - <start-run...>
End of run - <test-run...>
Start of a test suite - <start-suite...>
End of a test suite - <test-suite...>
Start of a test case - <start-test...>
End of a test case - <test-case...>

We'll probably just need to capture the "End of a test case" and report back from there to the same ingester we are using for build metrics.

In short we should be able to create a contactless "install nuget package and go" experience for the developers, for fast onboarding.

agoda-com / dotnet-build-metrics

Capturing Test run data from Developer Local Machines #25

Feature request

Type

Is your feature request related to a problem?

Describe the solution you'd like