Integration tests are slow in Azure DevOps

[simondel]

This project currently has one integration test. Running it takes about 30. In this integration test, Stryker.NET is executed on a project. This testrun is executed on the hosted agents by Microsoft. These are Standard_DS2_v2 machines with 2 vCPU's and 7 GiB memory.

Running this project using Stryker.NET from a NuGet dependency instead of from code is a lot faster (1 minute and 40 seconds). This test was also executed on a hosted agent by Microsoft.

The slow integration test has two specific points that are slow. You can see this in these logging statements:

2018-10-28T20:19:35.0895902Z [20:19:35 INF] Starting initial build
2018-10-28T20:34:51.8426672Z [20:34:51 DBG] Initial build output Microsoft (R) Build Engine version 15.8.166+gd4e8d81a88 for .NET Core

And:

2018-10-28T20:34:51.8435064Z [20:34:51 INF] Initial build successful
2018-10-28T20:49:53.7191937Z [20:49:53 VRB] "D:\a\1\s\integrationtests\IntegrationTests\Stryker.Core.IntegrationTest\ExampleProject\ExampleProject.XUnit dotnet msbuild ExampleProject.XUnit.csproj /nologo /t:PrintReferences" resulted in   ExampleClassLibrary -> D:\a\1\s\integrationtests\IntegrationTests\Stryker.Core.IntegrationTest\ExampleProject\ExampleClassLibrary\bin\Debug\netstandard2.0\ExampleClassLibrary.dll

The time between the logging statements is in both cases about 15 minutes. If you add this up, you get the 30 minute delay that we notice between the integration test and running the example project directly.

Since we're an open source project, we get 10 parallel pipelines. Each build uses up 3 of these pipelines (1 windows, 1 linux and 1 mac). This can cause our build queue to fill up quite easily.

[simondel]

Something that's worth testing out is using one of our own physical devices as a build agent. That way we can compare the performance between running the tests manually from the command line/Visual Studio and running them from the Build Agent code by Microsoft.