vstest.console hangs up when using parallel tests in combination with code coverage analysis

[N-Olbert]

Description

When using MSTestV2 in-assembly parallel test execution in combination with the CodeCoverage-DataCollector vstest.console.dll hangs up indefinitly.

The logs show that vstest.console.dll indefinitly pools (but gets no message): TpTrace Verbose: 0 : 20700, 12, 2019/08/29, 13:08:30.803, 16201432794, vstest.console.exe, TcpClientExtensions.MessageLoopAsync: Polling on remoteEndPoint: 127.0.0.1:3406 localEndPoint: 127.0.0.1:3405

Using dnspy and Windbg the problem could be traced down to the following line: https://github.com/microsoft/vstest/blob/7317f229d8844ef4826cf406e53d93bd7eb0ffd3/src/Microsoft.TestPlatform.CommunicationUtilities/SocketCommunicationManager.cs#L320

where the binary reader isn't initalized. Therefore a null reference execption will be thrown. After this Exception occured no callback to vstest.console.exe occurs anymore which causes vstest.console.exe to pool indefinitly. The Host-logs show this like this:

TpTrace Verbose: 0 : 17080, 33, 2019/08/29, 13:00:05.803, 14520945740, testhost.exe, TestExecutionRecorder.RecordStart: Starting test: MyNamespace.Mytest.
TpTrace Error: 0 : 17080, 33, 2019/08/29, 13:00:05.804, 14520948679, testhost.exe, TestCaseEventsHandler.RaiseTestCaseStart: Exception occurred while calling handler of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyOutOfProcDataCollectionManager for TestCaseStartEventArgs: System.Reflection.TargetInvocationException: Ein Aufrufziel hat einen Ausnahmefehler verursacht. ---> System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
   bei Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveRawMessage()
   bei Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveMessage()
   bei Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.DataCollectionTestCaseEventSender.SendTestCaseStart(TestCaseStartEventArgs e)
   --- Ende der internen Ausnahmestapelüberwachung ---
   bei System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   bei System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   bei System.Delegate.DynamicInvokeImpl(Object[] args)
   bei Microsoft.VisualStudio.TestPlatform.Utilities.MulticastDelegateUtilities.SafeInvoke(Delegate delegates, Object sender, EventArgs args, String traceDisplayName)

Steps to reproduce

The following runsettings file has been used:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <!-- Configurations that affect the Test Framework -->
  <RunConfiguration>
    <MaxCpuCount>1</MaxCpuCount>

    <!-- Path relative to solution directory -->
    <ResultsDirectory>.\TestResults</ResultsDirectory>

    <!-- x86 or x64 -->
    <!-- You can also change it from the top-level menu Test > Test Settings > Processor Architecture for AnyCPU Projects -->
    <TargetPlatform>x64</TargetPlatform>

    <!-- Specify timeout in milliseconds. A valid value should be greater than 0 -->
    <TestSessionTimeout>7200000</TestSessionTimeout>
  </RunConfiguration>

  <!-- Configurations for data collectors -->
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
                <ModulePath>.*Tests.dll</ModulePath>
                <ModulePath>.*OracleExtension.dll</ModulePath>
                <ModulePath>.*Plugins.*</ModulePath>
                <ModulePath>.*Newtonsoft.Json.dll</ModulePath>
              </Exclude>
            </ModulePaths>

            <!-- We recommend you do not change the following values: -->
            <UseVerifiableInstrumentation>true</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>true</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>true</CollectFromChildProcesses>
            <CollectAspDotNet>false</CollectAspDotNet>

          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

  <!-- MSTest adapter -->
  <MSTest>
    <MapInconclusiveToFailed>true</MapInconclusiveToFailed>
    <CaptureTraceOutput>true</CaptureTraceOutput>
    <DeleteDeploymentDirectoryAfterTestRunIsComplete>true</DeleteDeploymentDirectoryAfterTestRunIsComplete>
    <DeploymentEnabled>true</DeploymentEnabled>
  </MSTest>
</RunSettings>

Unfortunately I can't share the project as this is a business application. I havent created a demoproject because this error should be relatively easy to reproduce: Simply change the RevceiveRawMessage-method in SocketCommunicationManager.cs to the following:

        /// <summary>
        /// Reads message from the binary reader
        /// </summary>
        /// <returns> Raw message string </returns>
        public string ReceiveRawMessage()
        {
            lock (this.receiveSyncObject)
            {
                throw new NullReferenceException();
                // Reading message on binaryreader is not thread-safe
                return this.binaryReader.ReadString();
            }
        }

Expected behavior

No hang up until global test session timeout ocurs

Actual behavior

vstest.console.exe hangs up until global test session timeout occurs.

Environment

Windows 10 V 16299 vstest.console.exe V16.0.28916.169 MStestV2 (framework and adapter) 1.4.0

[brunmorten]

I'm experiencing the same issue on some of my builds.

[N-Olbert]

Mmmh... is this in fact defined behaviour?

"Features that rely on data collectors will need to globally turn OFF parallel execution. They can do so by either crafting a .runsettings file as shown above, or by passing the "--"syntax from the CLI. For e.g. the VSTest task with Test Impact Analysis ON will need to do this when invoking vstest runner." (https://github.com/Microsoft/testfx-docs/blob/master/RFCs/004-In-Assembly-Parallel-Execution.md)

If so, this is not a very good solution. At least the in-assembly parallel execution should be disabled automatically as soon as a data collector is specified. At the very least a warning should be shown.

[mayankbansal018]

@N-Olbert the adapters are not aware of whether data collections is enabled or not, & from vstest platform doesn't whether user intends to run tests in parallel within assembly as it can be achieved via adding adapter specific tags on test classes.

I understand this is a problem, but for now this is a design limitation

I'm closing this issue, please comment in case you have some concerns.

[DominiqueChabaud]

Hello @N-Olbert , I can see that you have 1 in your settings file. Does it solve the problem? If not how did you solve the problem?

[N-Olbert]

Hello @N-Olbert , I can see that you have 1 in your settings file. Does it solve the problem? If not how did you solve the problem?

Hi @DominiqueChabaud. Well… it is defined behavior, you can’t bypass it: Either you use Code Coverage (from MS) or parallel execution. Another option is to use Coverlet instead of MS-Code Coverage, according to the devs it should work fine with parallel execution (see https://github.com/coverlet-coverage/coverlet/issues/864).

The 1-setting has nothing to do with in-assembly parallel test execution, this setting specifies how many test assemblies are "really" executed in parallel, where as in-assembly parallel test execution is used to run multiple tests inside a single assembly concurrently.

[DominiqueChabaud]

Hello @N-Olbert. Thanks for your feedback. Our team did the same analysis and choose to deactivate MS Code Coverage and use Coverlet. Have a nice day.