Testing platform produces an error with dotnet test

[nbalu]

Describe the bug

Created a new test project with the new Microsoft.Testing.Platform which seems to be working with dotnet run command but not with dotnet test

Steps To Reproduce

Created a new NUnit test app: If I run dotnet run I get the following output:

C:\prog\NUnit.TempTests> dotnet test Test run for C:\prog\NUnit.TempTests\bin\Debug\net8.0\NUnit.TempTests.dll (.NETCoreApp,Version=v8.0) Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64) Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait... A total of 1 test files matched the specified pattern. Testhost process for source(s) 'C:\NUnit.TempTests\bin\Debug\net8.0\NUnit.TempTests.dll' exited with error: Error: An assembly specified in the application dependencies manifest (testhost.deps.json) was not found: package: 'Newtonsoft.Json', version: '13.0.1' path: 'lib/netstandard2.0/Newtonsoft.Json.dll' . Please check the diagnostic logs for more information.

Test Run Aborted.

---

If I run dotnet run I get the following output:

C:\prog\NUnit.TempTests> dotnet run .NET Testing Platform v1.3.1+359466b83c (UTC 2024. 07. 16.) [win-x64 - .NET 8.0.7] Passed! - Failed: 0, Passed: 5, Skipped: 0, Total: 5, Duration: 317ms - NUnit.TempTests.dll (win-x64 - .NET 8.0.7)

For some reason it seems like the Json reference flows through the test command which I'm not sure how it's supposed to be possible. The test itself is not using Newtonsoft.Json at all.

Expected behavior

Should produce the same run result and output.

Actual behavior

Throws the error above.

Additional context

I did made my tweaks to be able to make the NUnit-Vnext branch to be able to work. But that's not supposed to cause this error at all.

If you need any context / info please feel free to reach out to me.

Best, Bálint

[thomhurst]

Add the <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport> property to your .csproj

[Evangelink]

@nbalu please have a look at https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test for more informations.

We are aware this is a little cumbersome to have these different flags and maybe we should have them linked for NUnit (cc @OsirisTerje) but we had to have as many safeguards as possible when we have released this new platform. The number of edge cases or undocumented scenarios of VSTest is so high that we were sure to end up in some places where something would not work and we wanted to ensure there was always an easy way out for users.

[nbalu]

Thanks Guys, that was helpful.

And I can just imagine how much complexity all that legacy causes. Though right now I feel like I'm turning on flags that should be just simply be controlled by Microsoft.Testing.TestPlaform.

So people might be using:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.x" />
or
<PackageReference Include="Microsoft.Testing.Platform" Version="1.x" />

But not both thus flags could controlled by Platform as a shipped default props. As .net is great because of their sane defaults / no config hell. IMHO selecting the package could imply the flavour of the "runner". Or am I wrong here?

Running still not working as expected, got a wierd bug in the servicecollection dispose. I'll try to figure out what might be wrong with my setup reading through the docs pages. I do see that the log file is not created underneath the TestResults so I'll try to figure out what did go wrong here. Though this should not be happening.

C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests> dotnet test
  Determining projects to restore...
  All projects are up-to-date for restore.
  Microsoft.Testing.Platform.Extensions.TestRunner -> C:\prog\TestRunner\TestRunner\Microsoft.Testing.Platform.Extensions.TestRunner\bin\Debug\net8
  .0\Microsoft.Testing.Platform.Extensions.TestRunner.dll
  TestRunner.Agent.NUnit.TempTests -> C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTe
  sts.dll
  Run tests: 'C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll' [net8.0|x64]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed: Tests fail
ed: .NET Testing Platform v1.3.1+359466b83c (UTC 2024. 07. 16.) [win-x64 - .NET 8.0.7] [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTe
sts\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed: Unhandled
exception. System.ObjectDisposedException: The CancellationTokenSource has been disposed. [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.Tem
pTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Syst
em.Threading.CancellationTokenSource.get_Token() [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.c
sproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Micr
osoft.Testing.Platform.Services.CTRLPlusCCancellationTokenSource.get_CancellationToken() in /_/src/Platform/Microsoft.Testing.Platform/Services/CTR
LPlusCCancellationTokenSource.cs:line 25 [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Micr
osoft.Testing.Extensions.Telemetry.AppInsightsProvider.IngestLoopAsync() in /_/src/Platform/Microsoft.Testing.Extensions.Telemetry/AppInsightsProvi
der.cs:line 143 [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Micr
osoft.Testing.Platform.Helpers.TaskExtensions.TimeoutAfterAsync(Task task, TimeSpan timeout, Boolean observeException, String filePath, Int32 lineN
umber) in /_/src/Platform/Microsoft.Testing.Platform/Helpers/TaskExtensions.cs:line 121 [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempT
ests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Micr
osoft.Testing.Extensions.Telemetry.AppInsightsProvider.DisposeAsync() in /_/src/Platform/Microsoft.Testing.Extensions.Telemetry/AppInsightsProvider
.cs:line 324 [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Micr
osoft.Testing.Platform.Helpers.DisposeHelper.DisposeAsync(Object obj) in /_/src/Platform/Microsoft.Testing.Platform/Helpers/DisposeHelper.cs:line 2
5 [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Micr
osoft.Testing.Platform.Hosts.CommonTestHost.DisposeServiceProviderAsync(ServiceProvider serviceProvider, Func`2 filter, List`1 alreadyDisposed, Boo
lean isProcessShutdown) in /_/src/Platform/Microsoft.Testing.Platform/Hosts/CommonTestHost.cs:line 184 [C:\prog\TestRunner\TestRunner\TestRunner.Ag
ent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Micr
osoft.Testing.Platform.Hosts.CommonTestHost.RunAsync() in /_/src/Platform/Microsoft.Testing.Platform/Hosts/CommonTestHost.cs:line 56 [C:\prog\TestR
unner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Micr
osoft.Testing.Platform.Builder.TestApplication.RunAsync() in /_/src/Platform/Microsoft.Testing.Platform/Builder/TestApplication.cs:line 252 [C:\pro
g\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Test
ingPlatformEntryPoint.Main(String[] args) in C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\obj\Debug\net8.0\TestPlatformEntryPoint
.cs:line 19 [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:    at Test
ingPlatformEntryPoint.<Main>(String[] args) [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj
]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:  [C:\prog\
TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed: === COMMAN
D LINE === [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed: C:\Program
 Files\dotnet\dotnet.exe exec C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll
--internal-msbuild-node testingplatform.pipe.55bf509ac5d84be2988104264d00ffab  [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\Test
Runner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:  [C:\prog\
TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\bin\Debug\net8.0\TestRunner.Agent.NUnit.TempTests.dll : error run failed:  [C:\prog\
TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\Users\Timi\.nuget\packages\microsoft.testing.platform.msbuild\1.2.1\buildMultiTargeting\Microsoft.Testing.Platform.MSBuild.targets(248,5): error
 MSB4018: The "InvokeTestingPlatformTask" task failed unexpectedly. [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agen
t.NUnit.TempTests.csproj]
C:\Users\Timi\.nuget\packages\microsoft.testing.platform.msbuild\1.2.1\buildMultiTargeting\Microsoft.Testing.Platform.MSBuild.targets(248,5): error
 MSB4018: System.ArgumentNullException: Value cannot be null. (Parameter '_outputFileStream') [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit
.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\Users\Timi\.nuget\packages\microsoft.testing.platform.msbuild\1.2.1\buildMultiTargeting\Microsoft.Testing.Platform.MSBuild.targets(248,5): error
 MSB4018:    at Microsoft.Testing.Platform.Helpers.ArgumentGuard.IsNotNull(Object argument, String paramName) in /_/src/Platform/Microsoft.Testing.
Platform/Helpers/ArgumentGuard.cs:line 16 [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\Users\Timi\.nuget\packages\microsoft.testing.platform.msbuild\1.2.1\buildMultiTargeting\Microsoft.Testing.Platform.MSBuild.targets(248,5): error
 MSB4018:    at Microsoft.Testing.Platform.MSBuild.InvokeTestingPlatformTask.Execute() in /_/src/Microsoft.Testing.Platform.MSBuild/Tasks/InvokeTes
tingPlatformTask.cs:line 238 [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\Users\Timi\.nuget\packages\microsoft.testing.platform.msbuild\1.2.1\buildMultiTargeting\Microsoft.Testing.Platform.MSBuild.targets(248,5): error
 MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [C:\prog\TestRunner\TestRunner\TestR
unner.Agent.NUnit.TempTests\TestRunner.Agent.NUnit.TempTests.csproj]
C:\Users\Timi\.nuget\packages\microsoft.testing.platform.msbuild\1.2.1\buildMultiTargeting\Microsoft.Testing.Platform.MSBuild.targets(248,5): error
 MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingCon
text, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [C:\prog\TestRunner\TestRunner\TestRunner.Agent.NUnit.TempTests\Tes
tRunner.Agent.NUnit.TempTests.csproj]

And still dotnet run works as expected :) it also do not log into the testresults folder, though it runs the tests.

[Evangelink]

So people might be using:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.x" />
or
<PackageReference Include="Microsoft.Testing.Platform" Version="1.x" />

Having multiple packages is indeed a solution although it could not work as in your example.

One of the problem of VSTest, and current approach is that the test frameworks depends on some object model package shipped by VSTest but they don't declare this dependency and the user references Microsoft.NET.Test.Sdk that ships a version of this object model. Because of this design, the potential conflicts will happen at runtime and not compile time. This is also causing some issues where sometime people are not adding Microsoft.NET.Test.Sdk (because that's not super natural).

Instead, with the new platform, the user references only NUnit packages and there is no conflicting version of the new "object model". It would be possible for NUnit to continue shipping NUnit3TestAdapter as the old adapter (working only with VSTest) and to ship some new package working with the new platform.

One downside of this, and the reason we didn't do that on MSTest, is that it makes it harder for people to try it out. For example, right now it's relatively easy to keep all your infra has before and to occasionally (let's say only on your machine) run passing the flag enabling the runner. Again, we now have about 8 months of feedback on MSTest side and things are going really well so this is something that could totally be done now.

About the errors, I actually see 2 problems, one with dotnet test and one with dotnet exec. Can you confirm this log is from only one run? It seems that the version of Microsoft.Testing.Platform.MSBuild (1.2.1) is different from Microsoft.Testing.Platform (1.3.1). I'll investigate the ObjectDisposedException.

[nbalu2]

Yes it's from a single run.

My bad haven't checked that if a new version exists. I just had the time to play around with it after weeks/months maybe.

[Evangelink]

I found the issue for the object disposed exception but I will wait for @MarcoRossignoli on Monday to discuss best fix.

[Evangelink]

I think it's fixed by #3306, I'll cherry-pick and do a bug fix release of the platform.

[nbalu]

1.3.2 solved the execution problem and the output is working as expected thanks for the help👍