search

nunit / nunit-console

NUnit Console runner and test engine
MIT License
215 stars 152 forks source link

NUnit Console 3.15.2: Could not load file or assembly 'Microsoft.AspNetCore.Hosting.Abstractions' in WebApplicationFactory #1202

Open SuperRembo opened 2 years ago

SuperRembo commented 2 years ago

Hi,

I have a simple ASP.NET Core web app (source) with an integration test. It's basically a stripped down version of the Microsoft Integration tests in ASP.NET Core example.

With dotnet test the test passes, however the NUnit console runner fails with a FileNotFoundException.

Output:

❯ NUnit.ConsoleRunner.3.15.2\tools\nunit3-console.exe .\WebApi.Tests\bin\Debug\net6.0\WebApi.Tests.dll
NUnit Console 3.15.2 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Sunday, July 31, 2022 15:21:19

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    .\WebApi.Tests\bin\Debug\net6.0\WebApi.Tests.dll

Errors, Failures and Warnings

1) SetUp Error : WebApi.Tests.WebHostStartupFixture
System.IO.FileNotFoundException : Could not load file or assembly 'Microsoft.AspNetCore.Hosting.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1..ctor()
   at WebApi.Tests.WebHostStartupFixture..ctor() in C:\SuperRembo\AspNetIntegrationTests\WebApi.Tests\WebHostStartupFixture.cs:line 12

Run Settings
    DisposeRunners: True
    WorkDirectory: C:\SuperRembo\AspNetIntegrationTests
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETCoreApp,Version=v6.0
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    TargetRuntimeFramework: netcore-6.0
    NumberOfTestWorkers: 8

The output directory doesn't contain a Microsoft.AspNetCore.Hosting.Abstractions.dll, so apparently dotnet test has another way to find it.

Is this a bug in the console runner? Is there a way to tell the console runner where to find the assembly?

Note: There is a Microsoft.AspNetCore.Hosting.Abstractions package, but that hasn't been updated in a couple of years. There is no NET6 version available.

CharliePoole commented 2 years ago

I see you are using version 3.15.2 of the console runner and engine but are also referencing the NUnit 3 adapter version 4.2.1. That version of the adapter incorporates version 3.13.1 of the engine. This is probably not a problem because the console runner doesn't use the adapter at all, but it might clarify things if you temporarily removed the adapter reference. Alternatively, you could go back to version 3.13.1 of the console runner.

SuperRembo commented 2 years ago

The same error occurs without the adapter. dotnet test doesn't find any tests, as expected.

SuperRembo commented 2 years ago

Alternatively, you could go back to version 3.13.1 of the console runner.

Version 3.13.1 of the runner doesn't support NET 5.0 or NET 6.0 yet, so that doesn't help.

OAguinagalde commented 2 years ago

I believe this is the same issue I posted some days ago... It did take a some days to reach the same conclusion but if you read the last comment I wrote there it explains it, partially. https://github.com/nunit/nunit-console/issues/1208#issuecomment-1217547258

The issues seems to be that running the project via Visual Studio or dotnet, it relies on the ASP.Net core dependencies listed in this file Microsoft.AspNetCore.App.deps.json to find what it needs. However if running via nunit, that file is not used, so it cant locate dependencias listed there.

You can verify the issue is the same by checking the contents of the environment variable APP_CONTEXT_DEPS_FILES.

var allTheUsedDepsFiles = System.AppContext.GetData("APP_CONTEXT_DEPS_FILES");

After checking that variable, both running via VS and the nunit console this are the contents:

    Visual Studio
C:\\Users\\user\\git_projects\\MYSOLUTION\\ServiceTest\\bin\\Debug\\net5.0\\ServiceTest.deps.json
C:\\Program Files\\dotnet\\shared\\Microsoft.AspNetCore.App\\5.0.17\\Microsoft.AspNetCore.App.deps.json
C:\\Program Files\\dotnet\\shared\\Microsoft.NETCore.App\\5.0.17\\Microsoft.NETCore.App.deps.json

    nunit3-console:
C:\\Users\\user\\.nuget\\packages\\nunit.consolerunner\\3.15.2\\tools\\agents\\net5.0\\nunit-agent.deps.json;
C:\\Program Files\\dotnet\\shared\\Microsoft.NETCore.App\\5.0.17\\Microsoft.NETCore.App.deps.json
CharliePoole commented 2 years ago

I believe the fix for #1203 will also fix this but I'm leaving this one open pending confirmation.

@SuperRembo Can you try the 3.16.0-dev00043 release from our MyGet feed?

SuperRembo commented 2 years ago

Where can I find that version? I don't see it at https://www.myget.org/feed/nunit/package/nuget/NUnit. There 3.13.4-dev-07602 is the latest 3.x version

CharliePoole commented 2 years ago

You want the latest version of the NUnit console runner, not the NUnit framework. So, depending on which package you normally use, one of these...

SuperRembo commented 2 years ago

With the latest 3.x test runner I get a different error:

NUnit Console 3.16.0-dev00043 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Monday, October 17, 2022 14:26:54

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    .\WebApi.Tests\bin\Debug\net6.0\WebApi.Tests.dll

Errors, Failures and Warnings

1) Error : WebApi.Tests.WebHostStartupFixture.Should_get_content
System.InvalidOperationException : Can't find 'C:\SuperRembo\NUnit.ConsoleRunner.3.16.0-dev00043\tools\agents\net6.0\WebApi.deps.json'. This file is required for functional tests to run properly. There should be a copy of the file on your source project bin folder. If that is not the case, make sure that the property PreserveCompilationContext is set to true on your project file. E.g '<PreserveCompilationContext>true</PreserveCompilationContext>'. For functional tests to work they need to either run from the build output folder or the WebApi.deps.json file from your application's output directory must be copied to the folder where the tests are running on. A common cause for this error is having shadow copying enabled when the tests run.
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.EnsureDepsFile()
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.EnsureServer()
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(DelegatingHandler[] handlers)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(Uri baseAddress, DelegatingHandler[] handlers)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient(WebApplicationFactoryClientOptions options)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient()
   at WebApi.Tests.WebHostStartupFixture.Should_get_content() in C:\SuperRembo\AspNetIntegrationTests\WebApi.Tests\WebHostStartupFixture.cs:line 17
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.GetResult()
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
   at NUnit.Framework.Internal.Execution.SimpleWorkItem.<>c__DisplayClass4_0.<PerformWork>b__0()
   at NUnit.Framework.Internal.ContextUtils.<>c__DisplayClass1_0`1.<DoIsolated>b__0(Object _)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at NUnit.Framework.Internal.ContextUtils.DoIsolated(ContextCallback callback, Object state)
   at NUnit.Framework.Internal.ContextUtils.DoIsolated[T](Func`1 func)
   at NUnit.Framework.Internal.Execution.SimpleWorkItem.PerformWork()

Run Settings
    DisposeRunners: True
    WorkDirectory: C:\SuperRembo\AspNetIntegrationTests
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETCoreApp,Version=v6.0
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    TargetRuntimeFramework: netcore-6.0
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Failed
  Test Count: 1, Passed: 0, Failed: 1, Warnings: 0, Inconclusive: 0, Skipped: 0
    Failed Tests - Failures: 0, Errors: 1, Invalid: 0
  Start time: 2022-10-17 12:26:54Z
    End time: 2022-10-17 12:26:55Z
    Duration: 0.635 seconds

Results (nunit3) saved as TestResult.xml

The WebApi.deps.json does exist in the same folder as WebApi.Tests.dll, not in the `tools\agents\net6.0' folder.

CharliePoole commented 2 years ago

What happens if you copy WebApi.deps.json to the output folder?

SuperRembo commented 2 years ago

When I copy the WebApi.deps.json file from the WebApi output folder to the agent folder (tools/agents/net6.0), then it fails with

System.InvalidOperationException : Solution root could not be located using application root C:\SuperRembo\NUnit.ConsoleRunner.3.16.0-dev00043\tools\agents\net6.0\.
   at Microsoft.AspNetCore.TestHost.WebHostBuilderExtensions.UseSolutionRelativeContentRoot(IWebHostBuilder builder, String solutionRelativePath, String applicationBasePath, String solutionName)
   at Microsoft.AspNetCore.TestHost.WebHostBuilderExtensions.UseSolutionRelativeContentRoot(IWebHostBuilder builder, String solutionRelativePath, String solutionName)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.SetContentRoot(IWebHostBuilder builder)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.<ConfigureHostBuilder>b__22_0(IWebHostBuilder webHostBuilder)
   at Microsoft.Extensions.Hosting.GenericHostWebHostBuilderExtensions.ConfigureWebHost(IHostBuilder builder, Action`1 configure, Action`1 configureWebHostBuilder)
   at Microsoft.Extensions.Hosting.GenericHostWebHostBuilderExtensions.ConfigureWebHost(IHostBuilder builder, Action`1 configure)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.ConfigureHostBuilder(IHostBuilder hostBuilder)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.EnsureServer()
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(DelegatingHandler[] handlers)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(Uri baseAddress, DelegatingHandler[] handlers)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient(WebApplicationFactoryClientOptions options)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient()
   at WebApi.Tests.WebHostStartupFixture.Should_get_content() in C:\SuperRembo\AspNetIntegrationTests\WebApi.Tests\WebHostStartupFixture.cs:line 17
CharliePoole commented 2 years ago

I've tried a wide range of things on this one but it keeps looking for WebApi.deps.json in the agent folder. I think resolving this might take a detailed examination of the source code for the WebApplicationFactory among other things.

I'm marking it as "help wanted" and I'll drop it from the release if necessary.