microsoft / testfx

MSTest framework and adapter
MIT License
779 stars 259 forks source link

DynamicDataSource caching causes problems when running tests in parallel. #906

Closed Haplois closed 3 years ago

Haplois commented 3 years ago

After 2.2.4, as long as the values in data source are serializable, every row for the specific test case is considered a separate test case. If our assumptions break parallel test runs that's a regression. Is it possible for you to share a quick repro with us?

@Haplois I will share a complete project when I have more time, but in general, consider a test case like this:

[
  DataTestMethod,
  DynamicData(dynamicDataSourceName: nameof(GetData), DynamicDataSourceType.Method)
]
public async Task Create_Tasks_Requests_From_Invalid_Tasks_Definitions_Should_Return_BadRequest_Errors(Events.APIGatewayProxyRequest request, Requests.CreateTaskCommand.Arguments arguments)
{
    // test code
}

with GetData similar to this:

public static IEnumerable<object[]> GetData() =>
 new [] {
    new [] object {
         // initialize pair of objects
    },
    new [] object {
         // initialize pair of objects
    },
 }

If I run such test cases in parallel (method level parallelism) then I see all kinds of exception suggesting that the the test framework attempts to pull the enumerable on different threads. I have plenty of data sources structured like this:

public static IEnumerable<object[]> GetData()
{
    yield new InputData {
       // ...
    };

    yield new InputData {
       // ...
    };

    // etc.
}

These obviously can't be pulled in parallel either.

Originally posted by @paul-michalik in https://github.com/microsoft/testfx/issues/871#issuecomment-874630277

AB#1408168

Haplois commented 3 years ago

This class needs to be able work concurrently.

https://github.com/microsoft/testfx/blob/86845cf8d1d2d01148ad961e1b21e6b908d07d49/src/Adapter/MSTest.CoreAdapter/Helpers/DataSerializationHelper.cs#L14-L135

jr01 commented 3 years ago

@Haplois - just to add to this. I intermittently get this exception with 2.2.5:

  Message: 
    Object reference not set to an instance of an object.

  Stack Trace: 
    <>c__DisplayClass4_0.<GetSerializer>b__0(KeyValuePair`2 i)
    Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)
    DataSerializationHelper.GetSerializer(String typeName)
    DataSerializationHelper.Deserialize(String[] serializedData)
    TestMethodRunner.RunTestMethod()
    TestMethodRunner.Execute()

Changing the SerializerCache to a ConcurrentDictionary should fix this.

Haplois commented 3 years ago

Fixed by #998