The active test run was aborted. Reason: Test host process crashed

[xuzhg]

Description

For a defect reproducable from the vstest command line, describe the issue you've observed.

Steps to reproduce

1. git clone https://github.com/xuzhg/WebApi.git git checkout -b RoutingBaseForNEtCore30 origin/RoutingBaseForNEtCore30

2. dotnet --version

3. run the following command (please replace the following Folder): dotnet test [YOURLocalClonedFolder]\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\Build.AspNetCore3x\Microsoft.Test.E2E.AspNetCore3x.OData.csproj --logger trx --results-directory C:\BuildAgent\_work\_temp --configuration release --no-build -v diag

4. result:

Expected behavior

Shall pass all test cases running

Actual behavior

Crash with the error message:

    <RunInfos>
      <RunInfo computerName="ODataBuild50" outcome="Error" timestamp="2019-12-02T20:25:22.4166882+00:00">
        <Text>The active test run was aborted. Reason: Test host process crashed : Unhandled exception. Unhandled exception. System.Threading.Tasks.TaskCanceledException: The operation was canceled.&#xD;
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.TestDosAttack(String filter) in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 85&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.&lt;&gt;c__DisplayClass9_0.&lt;&lt;TestDosAttackWithMultipleThreads&gt;b__0&gt;d.MoveNext() in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 95&#xD;
--- End of stack trace from previous location where exception was thrown ---&#xD;
   at System.Threading.Tasks.Task.&lt;&gt;c.&lt;ThrowAsync&gt;b__139_1(Object state)&#xD;
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()&#xD;
   at System.Threading.ThreadPoolWorkQueue.Dispatch()&#xD;
System.Threading.Tasks.TaskCanceledException: The operation was canceled.&#xD;
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.TestDosAttack(String filter) in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 85&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.&lt;&gt;c__DisplayClass9_0.&lt;&lt;TestDosAttackWithMultipleThreads&gt;b__0&gt;d.MoveNext() in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 95&#xD;
--- End of stack trace from previous location where exception was thrown ---&#xD;
   at System.Threading.Tasks.Task.&lt;&gt;c.&lt;ThrowAsync&gt;b__139_1(Object state)&#xD;
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()&#xD;
   at System.Threading.ThreadPoolWorkQueue.Dispatch()&#xD;
</Text>
      </RunInfo>
    </RunInfos>

Environment

Please share additional details about the test environment.

only installed VS2019 Enterprise.

Additional Info

If i run the failure test case individually, it can pass:

as:

[singhsarab]

@xuzhg I am not able to reproduce the issue. I followed the steps you have shared.

[MarioAtTrimble]

We are getting this issue from time to time on our build server, too. It's not consistent and sometimes it works just fine. We didn't see before when we were running core 2.2, but now see when we are running 3.1

[nohwnd]

We were seeing similar issues in VS, there was a problem with cancellation of run that produced a cancellation error and was not caught in the correct place. It resulted in the same error, but probably not the same stack. That error is kind of a catch all for all errors when the other process disconnects in the middle of an operation so it is hard to tell. I would suggest upgrading to 16.5.0 to see if that was fixes.

[adaughterson]

We are using vstest indirectly (via Microsoft.NET.Test.Sdk), and I have upgraded it to 16.5.0, and yet the error persists.

> A total of 1 test files matched the specified pattern.
> The active test run was aborted. Reason: Test host process crashed : Fatal error. Internal CLR error. (0x80131506)
>    at System.Reflection.RuntimeModule.GetTypes(System.Reflection.RuntimeModule)
>    at System.Reflection.RuntimeAssembly.get_DefinedTypes()
>    at Xunit.Runner.VisualStudio.VsTestRunner.GetAvailableRunnerReporters(System.Collections.Generic.IEnumerable`1<System.String>)
>    at Xunit.Runner.VisualStudio.VsTestRunner+<>c__DisplayClass23_0.<GetRunnerReporter>b__0()
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ViaFactory(System.Threading.LazyThreadSafetyMode)
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExecutionAndPublication(System.LazyHelper, Boolean)
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CreateValue()
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].get_Value()
>    at Xunit.Runner.VisualStudio.VsTestRunner.GetRunnerReporter(LoggerHelper, Xunit.Runner.VisualStudio.RunSettings, System.Collections.Generic.IEnumerable`1<System.String>)
>    at Xunit.Runner.VisualStudio.VsTestRunner.RunTests(Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle, LoggerHelper, Xunit.Runner.VisualStudio.TestPlatformContext, Xunit.Runner.VisualStudio.RunSettings, System.Func`1<System.Collections.Generic.List`1<Xunit.Runner.VisualStudio.AssemblyRunInfo>>)
>    at Xunit.Runner.VisualStudio.VsTestRunner.Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor.RunTests(System.Collections.Generic.IEnumerable`1<System.String>, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle)
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.Utilities.LazyExtension`2<Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor,Microsoft.VisualStudio.TestPlatform.Common.Interfaces.ITestExecutorCapabilities>, System.Tuple`2<System.Uri,System.String>, Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Adapter.RunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle)
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(System.Collections.Generic.IEnumerable`1<System.Tuple`2<System.Uri,System.String>>, Int64)
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestsInternal()
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTests()
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.ExecutionManager.StartTestRun(System.Collections.Generic.Dictionary`2<System.String,System.Collections.Generic.IEnumerable`1<System.String>>, System.String, System.String, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ClientProtocol.TestExecutionContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ITestCaseEventsHandler, Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.ITestRunEventsHandler)
>    at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler+<>c__DisplayClass30_3.<OnMessageReceived>b__3()
>    at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler+<>c.<.ctor>b__18_1(System.Action)
>    at Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SafeProcessJob(System.__Canon)
>    at Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].BackgroundJobProcessor()
>    at Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.ctor>b__12_0()
>    at System.Threading.Tasks.Task.InnerInvoke()
>    at System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object)
>    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
>    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
>    at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
>    at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
>    at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
>    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
>    at System.Threading.ThreadHelper.ThreadStart(System.Object)

Are there any other recommendations?

[gregpakes]

Our builds have started failing with this error. Is there a way to get some decent logs out? We are using azure hosted pipelines.

We have been getting this error for months but it's been very intermittent, so hasn't bothered us. But over the past 2 days it has spiked hugely.

[amoerie]

I'm also seeing persistent test host crashes in a project that I contribute to.

Links:

• https://github.com/fo-dicom/fo-dicom/runs/861061964
• https://github.com/fo-dicom/fo-dicom/runs/857136803
• https://github.com/fo-dicom/fo-dicom/runs/857086214 (this was an internal CLR error)

[ChaosEngine]

In follow up to this I've created https://github.com/microsoft/vstest/issues/2501

[jesperoastrom]

We had the exact same issue with our build. It turned out to be misconfigured settings for code coverage. In our CodeCoverage.runsettings-file we were not properly excluding test projects from coverage.

[mcallaghan-bsm]

We have (possibly) the same intermittent issue here.

rough important list of packages in our UT test project:

        <PackageReference Include="Microsoft.NETCore.Platforms" Version="3.1.0" />
        <PackageReference Include="coverlet.collector" Version="1.3.0">
        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
        <PackageReference Include="xunit" Version="2.4.1" />
        <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1" />

Our default gitlab runner job is on a kubernetes runner, and defaults to docker image: "mcr.microsoft.com/dotnet/core/sdk:3.1-alpine"

Adding diag to a parallel job to see if we can repro with more diagnostics.

[plachta11b]

I have got here due to use of wrong function. Assert.ThrowsException<MyException>(async () => await myFunction()) Fixed by replacing to async version: await Assert.ThrowsExceptionAsync<MyException>(async () => await myFunction())

[chaoticsoftware]

Anyone has any update for this issue or any idea how to solve it?

[ghost]

I am seeing the same error message both with vstest.console.exe on Visual Studio 2019 16.8.4 and with dotnet test. My test projects are using MSTest V2 v2.1.0. On the Windows 10 machine (OS build 17763.1697) I have installed .NET framework 4.7.2.

dotnet test MyMsTestProject.dll --filter OneSpecificTest -v diag
Microsoft (R) Test Execution Command Line Tool Version 16.8.3
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
The active test run was aborted. Reason: Test host process crashed : Process is terminated due to StackOverflowException.

Test Run Aborted.

With dotnet test I get this error in the Application event log:

Faulting application name: testhost.net472.x86.exe, version: 15.0.0.0, time stamp: 0xd89b8ef9 Faulting module name: KERNELBASE.dll, version: 10.0.17763.1697, time stamp: 0x672c12bb Exception code: 0xc00000fd Fault offset: 0x00108f0d Faulting process id: 0x1b38 Faulting application start time: 0x01d6ea7f68419dc1 Faulting application path: C:\Program Files\dotnet\sdk\5.0.102\TestHost\testhost.net472.x86.exe Faulting module path: C:\Windows\System32\KERNELBASE.dll Report Id: 25d2099b-1d8a-4847-a90d-680e65d12f6e Faulting package full name: Faulting package-relative application ID:

With vstest.console.exe this error is logged:

Faulting application name: testhost.net472.exe, version: 15.0.0.0, time stamp: 0x89f6cab7 Faulting module name: clr.dll, version: 4.7.3740.0, time stamp: 0x5f7e5a1b Exception code: 0xc00000fd Fault offset: 0x000000000005c71b Faulting process id: 0x1a70 Faulting application start time: 0x01d6ea818c7cd9ec Faulting application path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\testhost.net472.exe Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll Report Id: 62f466eb-cece-41ac-a934-cf21ea4080c8 Faulting package full name: Faulting package-relative application ID:

A TestResults directory was created next to the test assembly, but it is completely empty.

Any idea how I can find the cause of the error?

[nohwnd]

@dbtfsbre you can install 5.0.101 dotnet SDK, and also install ProcDump so it is available in your PATH (if you open command line and type procdump, you should see output of the procdump tool). Then you do just:

dotnet test MyMsTestProject.dll --filter OneSpecificTest -v diag --blame-crash

And it should create a dump for you, open that in VS and you should see where that exception is coming from. Most likely it is your code calling some method recursively and forgetting to terminate.

[ghost]

@nohwnd Thanks. Something went wrong when copying files to the test machine for an investigation. The test is running fine in Visual Studio using Test Explorer and during our regular Release test runs. After compiling the test again directly on the test machine, I was able to run it with dotnet test as well as vstest.console.exe.

[mcallaghan-geotab]

FWIW: the solution we found was actually due to OOM kills from kubernetes gitlab-runner setup. The nodes were stretching their RAM too tightly, and eventually digging showed events where k8s was killing the dotnet process.

Looked like this:

Memory cgroup out of memory: Killed process 3010546 (dotnet) total-vm:5186748kB, anon-rss:378116kB, file-rss:67156kB, shmem-rss:0kB, UID:0 pgtables:1336kB oom_score_adj:936

This might not be the problem or solution for everyone, but wanted to share back as it is definitely something to consider!

[felixntny]

I'm having a similar problem. My actual test code, however, was a recursive function that would not exit the loop due to my test data. It also results in an infinite cycle. I adjusted my data to get out of the recursion function, and it worked.

[rkieslinger]

We had the exact same issue with our build. It turned out to be misconfigured settings for code coverage. In our CodeCoverage.runsettings-file we were not properly excluding test projects from coverage.

@jesperoastrom We have the same issue and it seems it is occurring since we re-activated code coverage. Can you please explain a little further, how you exactly solved your problem?

[jesperoastrom]

@rkieslinger If i remember correctly, we were running the test command and collecting coverage using the arguments: --collect "Code Coverage" --settings <SomePath>/CodeCoverage.runsettings. In the CodeCoverage.runsettings file we had misconfigured the exclusion of test modules/assemblies which caused this issue for us. We solved the problem by correctly exluding our test projects from code coverage using the CodeCoverage.runsettings file in ways described here: https://docs.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis#include-or-exclude-assemblies-and-members.

[demonccc]

We have (possibly) the same intermittent issue here.

rough important list of packages in our UT test project:

        <PackageReference Include="Microsoft.NETCore.Platforms" Version="3.1.0" />
        <PackageReference Include="coverlet.collector" Version="1.3.0">
        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
        <PackageReference Include="xunit" Version="2.4.1" />
        <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1" />

Our default gitlab runner job is on a kubernetes runner, and defaults to docker image: "mcr.microsoft.com/dotnet/core/sdk:3.1-alpine"

Adding diag to a parallel job to see if we can repro with more diagnostics.

Hi, have you fixed this issue? We have a similar problem with dotnet and gitlab + k8s.

[mcallaghan-geotab]

@demonccc - see https://github.com/microsoft/vstest/issues/2261#issuecomment-791823920

[demonccc]

@demonccc - see #2261 (comment)

Thanks for the response. Could you tell me what configuration have you changed in order to avoid de OOM killer? The k8s executor of Gitlab doesn't have this option, and you can't modify the OOM adj of kubernetes.

Thanks in advance.

[itaibh]

It happens to me too when converting a project to use .net 5. We use NUnit and I made sure I use all the latest versions of the libraries. Out of more than 700 tests in one of the suites, 7 throw an exception such as this which crashes the test host process. Before changing to .net 5 (from .net framework 4.6.1, which means many changes were required but in those specific tests the flow seems untouched) everything worked.

I debugged the old version and the new version simultaneously and I see the same exception getting thrown, but in the .net framework it's getting handled correctly and the test passes while in .net 5 the exception isn't handled by the test host process and it crashes.

The test itself is quite short but hides complexities regarding threads, which might shed some light on where the problem is. Since the old code made use of Thread.Abort() which was deprecated in favor of Task.Cancel(), we had to replace Threads with Tasks, which might also narrow the search for what the root cause of the problem is.

[itaibh]

Okay, I now have a reproduction for the problem, and it's thread related.

This is the error I get:

========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running selected tests in C:\Users\itaib\source\repos\TestHostCrashTest\TestHostCrashTest\bin\Debug\net5.0\TestHostCrashTest.dll
   NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Non-Explicit run
The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.Exception: some value
   at TestHostCrashTest.CrashTest.<>c.<CrashTestHost>b__0_0() in C:\Users\itaib\source\repos\TestHostCrashTest\TestHostCrashTest\CrashTest.cs:line 14
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

========== Test run aborted: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in < 1 ms ==========

and this is the code:

using NUnit.Framework;
using System.Threading;

namespace TestHostCrashTest
{
    public class CrashTest
    {
        [Test]
        public void CrashTestHost()
        {
            var t = new Thread(() =>
                throw new Exception("some value")
            );

            t.IsBackground = true;
            t.Start();
            t.Join();
        }
    }
}

[juwens]

It seems, that this is the default behavior of the .net runtime (except 1.0 and 1.1; see legacyUnhandledExceptionPolicy). I didn't knew that either. EVERY crashing Thread will terminate the application, even without a join.

There's not much vstest can do about it, i guess.

For very special scenarios you can use the mentioned flag in your app.config. This will prevent the app from terminating when an unhandled exception happens in a thread.

example app.config

<configuration>
   <runtime>
      <legacyUnhandledExceptionPolicy enabled="1"/>
   </runtime>
</configuration>

[itaibh]

@juwens Then how do you explain it works on .net 4.6? It's a different runtime... There's a specific problem introduced in .net core and above.

[juwens]

@juwens Then how do you explain it works on .net 4.6? It's a different runtime... There's a specific problem introduced in .net core and above.

just tried it with .net 4.5 target framework and it terminates the app too. Might be because clr tries to run code on the latest installed framework. which is 4.8 in my case.

my code:

using System;
using System.Threading;

namespace ConsoleAppNetFx
{
    internal class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("start netfx app");

            var t = new Thread(() =>
            {
                throw new Exception("exception in thread");
            });

            t.Start();

            while (true)
            {
                Console.WriteLine("sleep");
                Thread.Sleep(100);
            }

            Console.WriteLine("happy end");
        }
    }
}```

[itaibh]

@juwens Yes, of course it breaks a PROGRAM, but we're discussing a UNIT TEST

[juwens]

@juwens Yes, of course it breaks a PROGRAM, but we're discussing a UNIT TEST

are you aware, that a running test is just .net program (test host) which invokes your unit-test methods? 🤔 Of course it behaves the identical to a programm. There is no magic in the test host. Have you tried setting legacyUnhandledExceptionPolicy ?

[itaibh]

Yes, of course I'm aware of that. Are you aware that this test host is dynamically loading the tests from a DLL file? If it worked before then there's no reason it'll break now, which means this test host isn't doing it's job well. And I didn't try that since it's irrelevant for .net core and up, which doesn't even read app.config files anymore. Besides, setting the tests to run as STA isn't a good option since the code itself is multithreaded and we want to test in parallel as much as possible.

On Tue, Jan 18, 2022, 19:09 Jens @.***> wrote:

@juwens https://github.com/juwens Yes, of course it breaks a PROGRAM, but we're discussing a UNIT TEST

are you aware, that a running test is just .net program (test host) which invokes your unit-test methods? 🤔 Of course it behaves the identical to a programm. There is no magic in the test host. Have you tried setting legacyUnhandledExceptionPolicy ?

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vstest/issues/2261#issuecomment-1015628109, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABCNNYAWGXDILPNP5N6VHTUWWNDRANCNFSM4JU4X5FQ . You are receiving this because you commented.Message ID: @.***>

[brah-mcdude]

I had this issue. Here is how I resolved it:

1. I read the test logs (output | Tests tab).
2. I looked at the call-stack where the logs were saying that the process crashed.
3. It was an async void method. Obviously my code was causing the crash. Switched to async Task. Issue resolved.
4. But that was in my specific case. It should be something as simple as that in your specific case - in other words - your code is causing it.
5. If unable to resolve - post your Output | Tests somewhere and publish a link to it here.
6. We will help you figure it out. :)

[brah-mcdude]

I recommend closing this issue.

[mcallaghan-geotab]

@brah-mcdude - if the application/test code itself caused the crash, this should be much more obvious/different from the other scenarios presented "The active test run was aborted. Reason: Test host process crashed" <-- too generic.

[brah-mcdude]

@mcallaghan-geotab - Here is the message I was getting: "The active test run was aborted. Reason: Test host process crashed". My code caused it. I traced the call stack. Show me your code. I will show you the cause of this issue.

[cn-ml]

I just got a similar problem with my application in .NET 6

The active test run was aborted. Reason: Test host process crashed : Stack overflow.
Repeat 19163 times:
--------------------------------
   at Studip.NET.EntityHandlerBase`1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].get_BaseUrl()
--------------------------------

Edit: I just removed some lines of code including attributes/reflection and now the test runs perfectly. I don't know why that caused it and how to fix it. This is the line of code I removed:

typeof(TEntity)
        .GetCustomAttributes(typeof(SchemaNameAttribute), false)
        .OfType<SchemaNameAttribute>()
        .FirstOrDefault()?.Type
        ?? throw new InvalidProgramException($"Type '{typeof(TEntity).FullName}' has no '{nameof(SchemaNameAttribute)}' to be used in type handler base!");

[brah-mcdude]

@cn-ml - you have a "Stack overflow". Show us your call-stack.

[mamift]

I've been getting this same issue, but it seemingly occurs only when running a whole test project at once; if I try to test each individual test methods in their own session/run then I never encounter the problem.

I also never get a stack trace or any meaningful debug output just before the Stack overflow occurs.

This is the most I've been able to extract out of the test/debug output:

========== Starting test run ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET Core 3.1.26)
[xUnit.net 00:00:02.50]   Starting:    McrApi.Tests.Integration
The active test run was aborted. Reason: Test host process crashed : Stack overflow.

Running/debugging the test project seems to make no difference. I've never been able to get VS to even break on the Stack overflow; this is just bizarre.

Using Xunit/.NET Core 3.1.

[brah-mcdude]

"I've been getting this same issue, but it seemingly occurs only when running a whole test project at once; if I try to test each individual test methods in their own session/run then I never encounter the problem."

multiple tests are stress loading each other. try running each test with a cpu-stress app. https://learn.microsoft.com/en-us/sysinternals/downloads/cpustres.

or, maybe your tests are sharing data. maybe you have static (global) variables. scenario: test1 writes to global static variable1, test2 overwrites variable1. test1 reads a corrupted value from variable1.

[mamift]

maybe your tests are sharing data. maybe you have static (global) variables. scenario: test1 writes to global static variable1, test2 overwrites variable1. test1 reads a corrupted value from variable1.

Yeah I do have shared static variables, but their values are set in a static constructor and are read only.

What's bizarre here is the stack overflow error only occurs when I run the tests in VS or using the dotnet command. If I debug the tests in VS then it NEVER occurs. I can break in VS on other exceptions like ArgumentNullException or NullRef exceptions, but never stack overflow.

[brah-mcdude]

@mamift - having inconsistent test results is an indication of data corruption. is your static ctor invoked only once for ALL the tests? i would get rid of any static variables. each test should live in its own bubble universe. you are violating this sacred principle.

[brah-mcdude]

@mamift - if you have a multi-threaded code under test - that could also lead to inconsistent test results. but you would be able to reproduce the inconsistency in the test results even when you run a single test. so that is not a prime suspect.

[edit] use a cpu-stress-app https://learn.microsoft.com/en-us/sysinternals/downloads/cpustres. load up all cpu's to 100%. then run each test separately.

[srothkin]

After the most recent Visual Studio 2022 update I've started seeing this nearly constantly in one of our solutions (with 421 tests using xUnit). Using .NET 6. I've gone over this entire thread as well as some other discussions I've found. None lead to fixes. When I do get it to run without this error, there's only 1 failure (and that's a not implemented exception for a test I've yet to write).

[juwens]

After the most recent Visual Studio 2022 update I've started seeing this nearly constantly in one of our solutions (with 421 tests using xUnit). Using .NET 6. I've gone over this entire thread as well as some other discussions I've found. None lead to fixes. When I do get it to run without this error, there's only 1 failure (and that's a not implemented exception for a test I've yet to write).

try dotnet test —-blame it will reveal the source of the issue. In our case it were unhandled exceptions in async void methods and more tricky: async void lambdas.

[srothkin]

After the most recent Visual Studio 2022 update I've started seeing this nearly constantly in one of our solutions (with 421 tests using xUnit). Using .NET 6. I've gone over this entire thread as well as some other discussions I've found. None lead to fixes. When I do get it to run without this error, there's only 1 failure (and that's a not implemented exception for a test I've yet to write).

try dotnet test —-blame it will reveal the source of the issue. In our case it were unhandled exceptions in async void methods and more tricky async void lambdas.

How do I get Visual Studio to do that with the test explorer runner?

[JBoothUA]

just in case this helps someone, at a random time this line started randomly hanging our tests

_task = Task.Factory.StartNew(async () => { await Task.Delay((int)cleanInterval.TotalMilliseconds, _cancellationTokenSource.Token); --> while (!_cancellationTokenSource.Token.IsCancellationRequested)

[brah-mcdude]

I just took a look at the original issue's description. https://github.com/microsoft/vstest/issues/2261#issue-532187546 I looked at the crash call stack. And I noticed: TestDosAttackWithMultipleThreads. I would like to take a look at the code. Threads are very difficult to get right.

[brah-mcdude]

After the most recent Visual Studio 2022 update I've started seeing this nearly constantly in one of our solutions (with 421 tests using xUnit). Using .NET 6. I've gone over this entire thread as well as some other discussions I've found. None lead to fixes. When I do get it to run without this error, there's only 1 failure (and that's a not implemented exception for a test I've yet to write).

try dotnet test —-blame it will reveal the source of the issue. In our case it were unhandled exceptions in async void methods and more tricky async void lambdas.

That is literally what I wrote above as the cause of crash in my case (https://github.com/microsoft/vstest/issues/2261#issuecomment-1034018644). @xuzhg - please search for "async void" in your code.

[brah-mcdude]

just in case this helps someone, at a random time this line started randomly hanging our tests

_task = Task.Factory.StartNew(async () => { await Task.Delay((int)cleanInterval.TotalMilliseconds, _cancellationTokenSource.Token); --> while (!_cancellationTokenSource.Token.IsCancellationRequested)

@JBoothUA - With all due respect. your hanging line seems irrelevant to this issue. Not that this issue is actually an issue (I keep saying that there is no issue and we should close it). But your issue is even simpler than this issue. Both your issue and this issue seem to emanate from your (and @xuzhg's) bad threading code.

[jakubkrzysztofsikora]

After the most recent Visual Studio 2022 update I've started seeing this nearly constantly in one of our solutions (with 421 tests using xUnit). Using .NET 6. I've gone over this entire thread as well as some other discussions I've found. None lead to fixes. When I do get it to run without this error, there's only 1 failure (and that's a not implemented exception for a test I've yet to write).

try dotnet test —-blame it will reveal the source of the issue. In our case it were unhandled exceptions in async void methods and more tricky async void lambdas.

That is literally what I wrote above as the cause of crash in my case (#2261 (comment)). @xuzhg - please search for "async void" in your code.

Holy moly, that was it. Thanks dude.

[brah-mcdude]

@xuzhg can you please close this issue? Reason: CANNOT REPRODUCE.