[enhancement] Coverage Collection

[amirvenus]

Hi,

I have been using this fantastic library with xunit however, I have noted that the test coverage results are not collected/published.

I would be grateful if consideration is given to collecting test coverage data as part of the test execution so that they could be used in a CI/CD run.

Thanks!

[akoeplinger]

I assume you're looking at getting some .xml or other file produced by the coverage tooling from the device right? Which tool are you using? And on which platform?

[amirvenus]

I assume you're looking at getting some .xml or other file produced by the coverage tooling from the device right? Which tool are you using? And on which platform?

The idea is something like this: https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-code-coverage?tabs=linux or dotCover that in addition to the actual results of the tests (and stacktraces where applicable) it will also collect the code coverage i.e. covered/uncovered percentage/lines/statements etc.

I am using this tool on a macOS azdo self-hosted agent for a CI/CD pipeline.

Thanks!

[akoeplinger]

Yeah so there are essentially two steps: 1) plug the coverage collector into the test run 2) collect the coverage results xml/json.

The former is probably something you can do already in your runner class which is derived from iOSApplicationEntryPoint. The latter is a bit more complicated since there's no good way to get a file off of the device, we currently stream the nunit/xunit test results over the network: https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.TestRunners.Common/iOSApplicationEntryPointBase.cs#L19-L21

[amirvenus]

Actually I am using the TestRunners from the Maui repo with some slight modifications used with xharness in a CI/CD pipeline.

I only use net6.0-android (not maui) for my project so I have deleted anything non-android (iOS, macCatalyst, etc.)

I can see that they have this:

internal class HeadlessTestRunner : AndroidApplicationEntryPoint
{
    private readonly HeadlessRunnerOptions _runnerOptions;
    private readonly TestOptions _options;

    public HeadlessTestRunner(HeadlessRunnerOptions runnerOptions, TestOptions options)
    {
        _runnerOptions = runnerOptions;
        _options = options;

        var cache = Application.Context.CacheDir!.AbsolutePath;
        TestsResultsFinalPath = Path.Combine(cache, _runnerOptions.TestResultsFilename);
    }

    protected override bool LogExcludedTests => true;

    public override TextWriter? Logger => null;

    public override string TestsResultsFinalPath { get; }

    protected override int? MaxParallelThreads => System.Environment.ProcessorCount;

    protected override IDevice Device { get; } = new TestDevice();

    protected override IEnumerable<TestAssemblyInfo> GetTestAssemblies() =>
        _options.Assemblies
            .Distinct()
            .Select(assembly =>
            {
                // Android needs this file to "exist" but it uses the assembly actually.
                var path = Path.Combine(Application.Context.CacheDir!.AbsolutePath, assembly.GetName().Name + ".dll");
                if (!File.Exists(path))
                {
                    File.Create(path).Close();
                }

                return new TestAssemblyInfo(assembly, path);
            });

    protected override void TerminateWithSuccess() { }

    protected override TestRunner GetTestRunner(LogWriter logWriter)
    {
        var testRunner = base.GetTestRunner(logWriter);
        if (_options.SkipCategories?.Count > 0)
        {
            testRunner.SkipCategories(_options.SkipCategories);
        }

        return testRunner;
    }

    public async Task<Bundle> RunTestsAsync()
    {
        var bundle = new Bundle();

        TestsCompleted += OnTestsCompleted;

        await RunAsync();

        TestsCompleted -= OnTestsCompleted;

        if (File.Exists(TestsResultsFinalPath))
        {
            bundle.PutString("test-results-path", TestsResultsFinalPath);
        }

        if (bundle.GetLong("return-code", -1) == -1)
        {
            bundle.PutLong("return-code", 1);
        }

        return bundle;

        void OnTestsCompleted(object? sender, TestRunResult results)
        {
            var message =
                $"Tests run: {results.ExecutedTests} "
                + $"Passed: {results.PassedTests} "
                + $"Inconclusive: {results.InconclusiveTests} "
                + $"Failed: {results.FailedTests} "
                + $"Ignored: {results.SkippedTests}";

            bundle.PutString("test-execution-summary", message);

            bundle.PutLong("return-code", results.FailedTests == 0 ? 0 : 1);
        }
    }
}

Could you please advise how I can make it collect the coverage info (statements/lines/assembly,...) as well?

I actually find the 2nd option more plausible as I don't see why we can't be using the same tcpTextWriter to transfer the coverage results xml as well?!

[akoeplinger]

Could you please advise how I can make it collect the coverage info (statements/lines/assembly,...) as well?

That depends on the specific coverage tool, e.g. if it has an API to start/stop the collection then you could add that into RunTestsAsync()

I actually find the 2nd option more plausible as I don't see why we can't be using the same tcpTextWriter to transfer the coverage results xml as well?!

Yeah for Android it's much easier since we can pull arbitrary files from the device via adb. It would still need to be implemented in xharness so you can somehow communicate a file should be pulled. I'd be happy to accept a PR for that.

[amirvenus]

I think the natural choice would be this:

https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-code-coverage?tabs=linux XUnit.Coverlet.Collector

But I still don't know how to implement it in Xharness. Could you please provide some hints?

Thanks!

[akoeplinger]

I have no experience with coverlet, you'd need to ask them.

[amirvenus]

I have no experience with coverlet, you'd need to ask them.

I am more than happy to work on this and contribute to this project but I don't see where the dotnet test command is executed.

Could you please point me to the right direction which file(s) I should focus on first to achieve getting coverage result from a tool when running the tests?

Thanks!

[akoeplinger]

I am more than happy to work on this and contribute to this project but I don't see where the dotnet test command is executed.

We don't execute dotnet test ourselves since xharness just provides the xunit/nunit runner pieces, that probably happens in the DeviceTests.Runners in the maui repo.

[kotlarmilos]

At the moment, we don't plan on supporting this feature. Please feel free to reopen it if you plan to continue this effort and we will be happy to assist you.