experiment with running tests in parallel within some collections

[danmoseley]

Our unit tests in each test assembly are divided across various classes, usually (but certainly not always) chosen to group related tests together rather than for any regard for parallelism.

Xunit's default is that each class is its own test collection (we generally do not change this) and default parallelism is between test collections, but not within them (we do limit this in some cases example)

In some (many?) cases tests within a class would be happy to run in parallel with each other. Note that classes deriving from FileCleanupTestBase ought not to be unsafe for that reason alone, if it is careful to put each test in their own folder https://github.com/dotnet/runtime/blob/e555a0e16635c6ff2d88366f19d0781ac1fef0ee/src/libraries/Common/tests/TestUtilities/System/IO/FileCleanupTestBase.cs#L37

Suggestion: determine whether there are large categories of tests in some test assembly or other that we can identify could safely run concurrently with others in the same class, and experiment with enabling that to see whether it materially speeds up tests.

Ways to enable that

1. obviously, breaking into their own classes.
2. an extension point eg https://www.meziantou.net/parallelize-test-cases-execution-in-xunit.htm

[ghost]

Tagging subscribers to this area: @dotnet/area-meta See info in area-owners.md if you want to be subscribed.

Issue Details

---

Our unit tests in each test assembly are divided across various classes, usually (but certainly not always) chosen to group related tests together rather than for any regard for parallelism. Xunit's default is that each class is its own test collection (we generally do not change this) and default parallelism is between test collections, but not within them (we do limit this in some cases [example](https://github.com/dotnet/runtime/blob/e555a0e16635c6ff2d88366f19d0781ac1fef0ee/src/libraries/System.Diagnostics.Tracing/tests/BasicEventSourceTest/XUnitAssemblyAttributes.cs#L7)) In some (many?) cases tests within a class would be happy to run in parallel with each other. Note that classes deriving from FileCleanupTestBase ought not to be unsafe for that reason alone, if it is careful to put each test in their own folder https://github.com/dotnet/runtime/blob/e555a0e16635c6ff2d88366f19d0781ac1fef0ee/src/libraries/Common/tests/TestUtilities/System/IO/FileCleanupTestBase.cs#L37 Suggestion: determine whether there are large categories of tests in some test assembly or other that we can identify could safely run concurrently with others in the same class, and experiment with enabling that to see whether it materially speeds up tests. Ways to enable that 1. obviously, breaking into their own classes. 2. an extension point eg https://www.meziantou.net/parallelize-test-cases-execution-in-xunit.htm

Author:	danmoseley
Assignees:	-
Labels:	`area-Meta`
Milestone:	-

[danmoseley]

I have no particular evidence there are interesting gains to be had here, just logging in case someone in the community is potentially interested in looking at this kind of thing, or opportunities for speeding up our unit tests.

[ericstj]

Might be worthwhile to take a look at which tests take the longest to run and might benefit from such parallelization.

@danmoseley did you happen to notice any case where we had a particularly long running test project and that was due to a test class with a ton of tests? cc @dotnet/area-system-io @dotnet/area-system-text-json

[stephentoub]

I have no particular evidence there are interesting gains to be had here, just logging in case someone in the community is potentially interested in looking at this kind of thing, or opportunities for speeding up our unit tests.

Might be worthwhile to take a look at which tests take the longest to run and might benefit from such parallelization.

Yes, I'm not interested in fielding PRs that just move stuff around and change the structure from what the dev originally created just because.

If there's a particular suite taking "too long", we decide it's worthwhile improving, and splitting it up is found to be the right solution, then great; that's about finding/noticing and fixing a real problem. Such cases are frequently better solved in other ways, though. For example, we've had cases in the past where reliance on theories yielded hundreds of thousands of test cases, each of which with overhead, and the fix was to move some of the theories to just be loops in a single case. Or cases with source generation where the fix was to batch what we sent to Roslyn.

Starting from a possible solution and looking for problems to which we can apply it is a bit backwards.

[adamsitnik]

I would be happy to spend literally few hours to implement https://github.com/xunit/xunit/issues/2484 to get proper tooling support for diagnosing such isssues

[danmoseley]

Starting from a possible solution and looking for problems to which we can apply it is a bit backwards.

The problem is that I have to wait while unit tests run; I don't see this as looking for a problem. Certainly it may not be a useful way to improve that. It sounds like Adam might have a cheap way to see whether it is.

[stephentoub]

xunit already logs data about test execution. I've used this to analyze it in the past:

using System.Linq;
using System.Xml;

string testResultsPath = @"D:\repos\runtime\artifacts\bin\System.Text.RegularExpressions.Tests\Debug\net9.0\testResults.xml";
string testTypeFilter = ""; // to filter down to a specific test class
bool groupResultsByMethod = false; // if false, groups by test class; if true, by method... true is helpful when looking at theories
bool printGroupedTests = false; // to print out the individual tests in each group

var doc = new XmlDocument();
doc.Load(testResultsPath);

var results = doc
    .SelectNodes("/assemblies/assembly/collection/test").Cast<XmlNode>()
    .Where(node => node.Attributes["type"].Value.Contains(testTypeFilter))
    .Select(node => new { TestName = node.Attributes["name"].Value, ContainingType = node.Attributes["type"].Value, TestTime = double.Parse(node.Attributes["time"].Value) })
    .GroupBy(a => groupResultsByMethod ? WithoutArgs(a.TestName) : a.ContainingType)
    .Select(g => new { TestType = g.Key, TotalTests = g.Count(), TotalTime = g.Sum(v => v.TestTime), Group = g })
    .OrderByDescending(v => v.TotalTime);

foreach (var g in results)
{
    Console.WriteLine($"{g.TestType} ({g.TotalTests}): {g.TotalTime}");
    if (printGroupedTests)
    {
        foreach (var test in g.Group.OrderByDescending(t => t.TestTime))
            Console.WriteLine($"\t{test.TestName}: {test.TestTime}");
    }
}

static string WithoutArgs(string name)
{
    int i = name.IndexOf('(');
    return i == -1 ? name : name.Substring(0, i);
}