Parallel tests execution in a collection

[iikuzmychov]

I have a very common case of testing. I have a collection fixture:

[CollectionDefinition(nameof(SomeCollection))]
public class SomeCollection : ICollectionFixture<ImmutableClass>
{
}

with some test classes:

[Collection(nameof(SomeCollection))]
public class TestClass1
{
    [Fact]
    public async Task Test1() => await Task.Delay(3000);

    [Fact]
    public async Task Test2() => await Task.Delay(3000);
}

[Collection(nameof(SomeCollection))]
public class TestClass2
{
    [Fact]
    public async Task Test3() => await Task.Delay(3000);

    [Fact]
    public async Task Test4() => await Task.Delay(3000);
}

This fixture is immutable, so I want to run each test method in the collection parallel. It should be 4 times faster. Is there any way to achieve this?

[HasanAouzoun]

@iikuzmychov I've opened a PR for this request. https://github.com/meziantou/Meziantou.Xunit.ParallelTestFramework/pull/12

[HasanAouzoun]

@iikuzmychov the feature that you are looking for is now available. you will have to add "EnableParallelization" attribute to the collection tests classes.

[Collection(nameof(SomeCollection))]
[EnableParallelization]
public class TestClass1
{
    [Fact]
    public async Task Test1() => await Task.Delay(3000);

    [Fact]
    public async Task Test2() => await Task.Delay(3000);
}

[Collection(nameof(SomeCollection))]
[EnableParallelization]
public class TestClass2
{
    [Fact]
    public async Task Test3() => await Task.Delay(3000);

    [Fact]
    public async Task Test4() => await Task.Delay(3000);
}

[iikuzmychov]

@HasanAouzoun awesome, thanks! Will try it soon & close the issue

[iikuzmychov]

Hmm, it seems like tests from TestClass1 and TestClass2 still executing sequentially 🤔 However, the tests within the same test class are running in parallel

[pregress]

I guess this is because of:

In XUnit.TestCollectionRunner

 protected virtual async Task<RunSummary> RunTestClassesAsync()
 {
     var summary = new RunSummary();

     foreach (var testCasesByClass in TestCases.GroupBy(tc => tc.TestMethod.TestClass, TestClassComparer.Instance))
     {
         summary.Aggregate(await RunTestClassAsync(testCasesByClass.Key, (IReflectionTypeInfo)testCasesByClass.Key.Class, testCasesByClass));
         if (CancellationTokenSource.IsCancellationRequested)
             break;
     }

     return summary;
 }

I think it should be implemented with Task.WaitAll() instead of summary.Aggregate(await

I'll launch a PR for this.

[pregress]

@meziantou Could you review the PR?

[pregress]

@meziantou can you release the nuget package and close this ticket?

[meziantou]

I just published the package on nuget. It should be available in a few minutes.

[iikuzmychov]

@meziantou @pregress @HasanAouzoun Thank you both!

Note for everyone who will found this issue in the future: now you also need to add [EnableParallelization] attribute to the fixture. So the final sample will looks like this:

public class ImmutableClass
{
}

[CollectionDefinition(nameof(SomeCollection))]
[EnableParallelization] // <------------------------------------------------  here is it
public class SomeCollection : ICollectionFixture<ImmutableClass>
{
}

[Collection(nameof(SomeCollection))]
[EnableParallelization]
public class TestClass1
{
    [Fact]
    public async Task Test1() => await Task.Delay(3000);

    [Fact]
    public async Task Test2() => await Task.Delay(3000);
}

[Collection(nameof(SomeCollection))]
[EnableParallelization]
public class TestClass2
{
    [Fact]
    public async Task Test3() => await Task.Delay(3000);

    [Fact]
    public async Task Test4() => await Task.Delay(3000);
}