ClassData/MemberData not being parallelized if reference type is being yielded

[aomader]

I do not know whether there is a specific reason for that, but as soon as a theory's data contains reference types (i.e., class objects), the theory is not run in parallel, but sequentially.

Consider for example the following two tests:

public class Test
{
    [Theory]
    [ClassData(typeof(TestInts))]
    public Task DoesRunInParallel(int i) => Task.Delay(2000);

    [Theory]
    [ClassData(typeof(TestObjs))]
    public Task DoesNotRunInParallel(Obj o) => Task.Delay(2000);
}

public class TestInts : TheoryData<int>
{
    public TestInts()
    {
        Add(1);
        Add(1);
        Add(1);
        Add(1);
        Add(1);
    }
}

public class Obj {}

public class TestObjs : TheoryData<Obj>
{
    public TestObjs()
    {
        Add(new Obj());
        Add(new Obj());
        Add(new Obj());
        Add(new Obj());
        Add(new Obj());
    }
}

The test DoesRunInParallel in fact does run in parallel, as advertised. However, the test DoesNotRunInParallel is run sequentially, while the only difference between those two is that the latter test's data yields a reference type (Obj).

Is there specific reason why that is?

[aomader]

Turns out the IXunitSerializable is necessary for that on the data.

[cmeeren]

This is not (just) about value vs. reference types. I have a custom struct that also has this behavior.

Furthermore, when investigating IXunitSerializable, it seems that it's mostly used to make xUnit report multiple test cases instead of just a single test case. However, in my case, xUnit successfully reports multiple test cases even without having to implement IXunitSerializable. So I'm not sure why the tests are not parallel, too.

Is there any reason why this framework should require IXunitSerializable for parallelization, when xUnit does not seem to require it to report each of the test cases separately?

Edit: Note that I'm running the tests in Rider, so it's possible that Rider is smarter than plain xUnit with regards to reporting individual test cases.

[meziantou]

@cmeeren Can you provide a simple repro?

[cmeeren]

Repro project attached:

Repro.zip

For quick reference, here's the code:

namespace Repro;

public struct MyStruct
{
    public int Data { get; }

    public MyStruct(int data)
    {
        Data = data;
    }

    public override string ToString()
    {
        return Data.ToString();
    }
}

public class UnitTest1
{
    public static IEnumerable<object[]> GetCustomStructTestData()
    {
        yield return new object[] {new MyStruct(1)};
        yield return new object[] {new MyStruct(2)};
    }

    [Theory]
    [MemberData(nameof(GetCustomStructTestData))]
    public void DoesNotRunInParallel(MyStruct data)
    {
        Thread.Sleep(2000);
    }

    public static IEnumerable<object[]> GetPrimitiveTestData()
    {
        yield return new object[] {1};
        yield return new object[] {2};
    }

    [Theory]
    [MemberData(nameof(GetPrimitiveTestData))]
    public void RunsInParallel(int data)
    {
        Thread.Sleep(2000);
    }
}

And here is what I see when I run it in Rider:

https://github.com/meziantou/Meziantou.Xunit.ParallelTestFramework/assets/7766733/e87f35ea-714e-4d83-bb3c-d67c7a97a3f6

Again though, it may be that Rider is smarter than VS, because here's what I see in VS - note that the test that does not run in parallel, is listed as a single test:

In any case, would be interesting to know why Rider manages this. Could perhaps indicate that there is a way to do this without having to implement IXunitSerializable.

[meziantou]

xunit exposes only one test when the struct doesn't implement IXunitSerializable. This test run all test cases sequentially. So, the ParallelTestFramework cannot parallelize test cases as it is not aware that there are multiple test cases.

[cmeeren]

I see. Then the fact that I see multiple test cases in Rider is something that Rider detects itself.

[meziantou]

Rider/VS can have their own logic to detect tests. The idea is to avoid using the test protocol, which requires to build the code, to speed up the UI when they can detect tests from the code.

I wonder if Rider is able to run a single test case, as it doesn't seem to be supported by xUnit.

[cmeeren]

I wonder if Rider is able to run a single test case, as it doesn't seem to be supported by xUnit.

Yes, I can confirm that this works (regardless of whether or not the data is serializable).