search

microsoft / testfx

MSTest framework and adapter
MIT License
749 stars 255 forks source link

Test Explorer fails to show tests. Tests output: "The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)" #241

Closed daiplusplus closed 2 years ago

daiplusplus commented 7 years ago

Description

I cloned a private Git repo on my local machine. The solution comprises of several projects all targeting .NET 4.6.2. I built the solution which pulled-in some NuGet packages, including MSTest.TestAdapter, 1.1.18 and MSTest.TestFramework, 1.1.18. There is a test project in the solution with several [TestClass]/[TestMethod] tests. The solution builds successfully, however the Test Explorer window is empty. On a coworker's machine running VS2015 the Test Explorer is populated correctly.

In the Output window's Test output I see the following:

------ Discover test started ------ The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047) ========== Discover test finished: 0 found (0:00:00.1886778) ==========

Steps to reproduce

I have no specific reproduction steps - it happened spontaneously on my machine. Restarting VS, cleaning the project output and TestResults directory, and closing+reopening the Test Explorer window has no effect.

Expected behavior

The tests in the project should appear in Test Explorer.

Actual behavior

No tests appear in Test Explorer, and an error message in the Output window.

Environment

Please share additional details about the test environment. Operating system, Build version of vstest.console, Package version of MSTest framework and adapter

AbhitejJohn commented 7 years ago

@jehoel: Sorry you are running into this. Few queries:

  1. Are the test projects targeting x64 architecture? If yes, is Test -> Test Settings -> Default Process Architecture set to x64?

  2. Can you set a system wide environment variable VS_UTE_Diagnostics to 1, restart VS and run through the scenario again please? This will output diagnostic logs in Output -> Tests pane in VS. Please do share that.

    Edit by @smadala See following two comments.

https://github.com/Microsoft/testfx/issues/241#issuecomment-375278449 https://github.com/Microsoft/testfx/issues/241#issuecomment-378622443

daiplusplus commented 7 years ago

  1. All projects are configured to build for AnyCPU. The "Prefer 32-bit" checkbox is greyed out. 1.1. I changed "Default process architecture" to "x64" to see what would happen, and after building I see this in the Test Output window:

    ------ Discover test started ------
Test run will use DLL(s) built for framework Framework45 and platform X64. Following DLL(s) will not be part of run: 
MyAssembly.exe is built for Framework Framework45 and Platform X86.
 Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)
========== Discover test finished: 0 found (0:00:00.2255871) ==========

    (MyAssembly.exe is a 4.6.2 AnyCPU project that does have "Prefer 32-bit" checked) (Interesting that it says Framework45 instead of Framework461, or is that irrelevant?)

  2. I opened the VS2017 command-prompt and did this:

    **********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.0.26430.16
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
C:\Program Files (x86)\Microsoft Visual Studio 15.0>SET VS_UTE_Diagnostics=1
C:\Program Files (x86)\Microsoft Visual Studio 15.0>devenv

    When VS opened and I rebuilt the solution, there was no additional output in the Test Output window, only the same message as before:

    ------ Discover test started ------
The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)
========== Discover test finished: 0 found (0:00:00.19589) ==========
AbhitejJohn commented 7 years ago

Thanks @Jehoel.

  1. Yes, Framework45 is irrelevant.
  2. You would need to set it for all applications, run 
    setx VS_UTE_Diagnostics 1

    Looking at your issue again and the fact that this is working in VS2015, I suspect this could be because of a missing assembly(possibly in GAC in VS2015). Could you please turn on fusion logs and look for any assembly load failures? Here is how you could turn them on. You could zip the logs and share them over here as well.

daiplusplus commented 7 years ago

I now see more output in the Test window, it contains data I'd rather not publish publicly - can I message you on Skype-for-Business?

Here's the tail end of the output that seems the most relevant - no obvious errors though:

PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile -

(etc - including more PowerShellTestContainerDiscoverer:IsTestFile - lines, some of which have valid filenames, others are empty)

PowerShellTestContainerDiscoverer:OnTestContainersChanged - FindPs1Files
test container discoverer executor://powershelltestexecutor/v1, discovered 0 containers
No containers found from 'PowerShellTools.TestAdapter.PowerShellTestContainerDiscoverer' :
test container discoverer executor://nodejstestexecutor/v1, discovered 0 containers
No containers found from 'Microsoft.NodejsTools.TestAdapter.TestContainerDiscoverer' :
test container discoverer executor://orderedtestadapter/v1, discovered 0 containers
No containers found from 'Microsoft.VisualStudio.MSTest.TestWindow.OrderedTestContainerDiscoverer' :
test container discoverer executor://generictestadapter/v1, discovered 0 containers
No containers found from 'Microsoft.VisualStudio.MSTest.TestWindow.GenericTestContainerDiscoverer' :
test container discoverer executor://webtestadapter/v1, discovered 0 containers
No containers found from 'Microsoft.VisualStudio.MSTest.TestWindow.WebTestContainerDiscoverer' :
DiscoveryOperation<DiscoverAllOrRunOnInitializeOperation> FinishedChangedCotainers, changed container count is 7
VirtualReadOnlyTestDataStore.OperationStateChanged State=ChangeDetectionFinished, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=ChangeDetectionFinished, InProgress=False
Discovering the following containers :
    C:\Git\MyProject\MyProject.CloudService\MyProject.FooWorkerRole\bin\Debug\MyProject.FooWorkerRole.dll
    C:\Git\MyProject\MyProject.Bar\bin\Debug\MyProject.Bar.exe
    C:\Git\MyProject\MyProject.Baz.Packager\bin\Debug\MyProject.Baz.Packager.dll
    C:\Git\MyProject\MyProject.Baz.Tests\bin\Debug\MyProject.Baz.Tests.dll
    C:\Git\MyProject\MyProject.Baz\bin\Debug\MyProject.Baz.dll
    C:\Git\MyProject\MyProject.SharedConfiguration\bin\Debug\MyProject.SharedConfiguration.dll
    C:\Git\MyProject\MyProject.StorageCli\bin\Debug\MyProject.StorageCli.exe
VirtualReadOnlyTestDataStore.OperationStateChanged State=DiscoveryStarting, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=DiscoveryStarting, InProgress=True
------ Discover test started ------
VirtualReadOnlyTestDataStore.OperationStateChanged State=DiscoveryStarted, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=DiscoveryStarted, InProgress=True
***    Discover TestCase using 'InMemoryUnitTestWriter' ***
RunSettings Content:
<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>C:\Git\MyProject\TestResults</ResultsDirectory>
    <SolutionDirectory>C:\Git\MyProject\</SolutionDirectory>
    <TargetPlatform>X86</TargetPlatform>
    <TargetFrameworkVersion>Framework45</TargetFrameworkVersion>
  </RunConfiguration>
</RunSettings>
The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)
========== Discover test finished: 0 found (0:00:57.5407445) ==========
VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetFinished, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=OperationSetFinished, InProgress=False
    Provider 'GroupByClassProvider' found 0 groups.
VirtualReadOnlyTestDataStore.OperationStateChanged State=DiscoveryFinished, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=DiscoveryFinished, InProgress=False
Pausing Queue..., current operation is: no current operation
About to Enqueue operation 'WaitForBuildOperation', hashcode:34282735 
Enqueue operation 'WaitForBuildOperation', hashcode:34282735 
Operation left in the the queue: 1
    'WaitForBuildOperation', hashcode:34282735

Processing Queue .....
Operation Dequeue : 'WaitForBuildOperation'
VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetStarted, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=OperationSetStarted, InProgress=False
Firing BuildCompleted event...
Container Discoverer 'Microsoft.VisualStudio.TestWindow.VsAdapters.VsProjectOutputContainerDiscoverer' has container changes
VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetFinished, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=OperationSetFinished, InProgress=False
test container discoverer executor://vsprojectoutputcontainerdiscoverer/v1, discovered 7 containers
Containers from 'Microsoft.VisualStudio.TestWindow.VsAdapters.VsProjectOutputContainerDiscoverer' :
    C:\Git\MyProject\MyProject.CloudService\MyProject.FooWorkerRole\bin\Debug\MyProject.FooWorkerRole.dll:executor://vsprojectoutputcontainerdiscoverer/v1
    C:\Git\MyProject\MyProject.Bar\bin\Debug\MyProject.Bar.exe:executor://vsprojectoutputcontainerdiscoverer/v1
    C:\Git\MyProject\MyProject.Baz.Packager\bin\Debug\MyProject.Baz.Packager.dll:executor://vsprojectoutputcontainerdiscoverer/v1
    C:\Git\MyProject\MyProject.Baz.Tests\bin\Debug\MyProject.Baz.Tests.dll:executor://vsprojectoutputcontainerdiscoverer/v1
    C:\Git\MyProject\MyProject.Baz\bin\Debug\MyProject.Baz.dll:executor://vsprojectoutputcontainerdiscoverer/v1
    C:\Git\MyProject\MyProject.SharedConfiguration\bin\Debug\MyProject.SharedConfiguration.dll:executor://vsprojectoutputcontainerdiscoverer/v1
    C:\Git\MyProject\MyProject.StorageCli\bin\Debug\MyProject.StorageCli.exe:executor://vsprojectoutputcontainerdiscoverer/v1
Container discoverer 'Microsoft.VisualStudio.TestWindow.VsAdapters.VsProjectOutputContainerDiscoverer' has no containers
AbhitejJohn commented 7 years ago

Thanks @Jehoel. As I suspected, these logs do not seem to be giving us much. Could you turn on fusion logging instead following the instructions above.

Looking at your issue again and the fact that this is working in VS2015, I suspect this could be because of a missing assembly(possibly in GAC in VS2015). Could you please turn on fusion logs and look for any assembly load failures? Here is how you could turn them on. You could zip the logs and share them over here as well

On

can I message you on Skype-for-Business?

Sure, you can send a mail to vstestsup@microsoft.com with any other details you might have.

StingyJack commented 7 years ago

@AbhitejJohn - what was the resolution for this?

Half of my tests are getting ignored in both VS and TFS.

daiplusplus commented 7 years ago

I generated the fusion logs - I had a quick look but I couldn't see any errors.

I do have another test project that works fine and its tests do appear in the Test Explorer window - so I compared the two projects. I noticed they're both .NET Framework 4.6.2 projects, however the working project has a reference to Microsoft.VisualStudio.QualityTools.UnitTestFramework which is being pulled from C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll - whereas the non-working project has a reference to Microsoft.VisualStudio.TestPlatform.TestFramework which is being pulled from (solution root)\packages\MSTest.TestFramework.1.1.18\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.dll.

Indeed, the working project is using a standard Visual Studio Test project which came with its own VS-coupled references, while the non-working project is using the NuGet "MSTest V2" assemblies instead.

So why is MSTest V2 not working all of a sudden?

TL;DR Workaround:

  1. Remove the NuGet package references and project assembly references to Microsoft.VisualStudio.TestPlatform.TestFramework.
  2. Replace them with a reference to your Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll assembly instead, which is located inside your Visual Studio installation directory (Common7\IDE\PublicAssemblies).
daiplusplus commented 7 years ago

I've emailed vstestsup@microsoft.com with my fusion logs.

daiplusplus commented 7 years ago

My email to vstestsup@microsoft.com was rejected by the Microsoft e-mail server with this message:

Your message to VSADepT@microsoft.com couldn't be delivered.

The group VSADepT only accepts messages from people in its organization or on its allowed senders list, and your email address isn't on the list.

I see that VSADepT is not the same thing as VSTestSup - but I'm not sure if the intended recipients receive it or not - @AbhitejJohn did you get it?

josteink commented 7 years ago

TL;DR Workaround:

That may work locally, but won't fly on our CI server. A real proper resolution would be real nice :(

josteink commented 7 years ago

I found the reason for the failure on my end.

Basically some of my projects are really old, and have slowly been migrated to newer .NET-versions and test-frameworks.

Turns out that the app.config in my project I had the following section still having around, despite the project being "upgraded" to .NET 4.5.1 and latest MSTest:

  <system.codedom>
    <compilers>
      <compiler language="c#;cs;csharp" extension=".cs" type="Microsoft.CSharp.CSharpCodeProvider, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" warningLevel="4">
        <providerOption name="CompilerVersion" value="v3.5"/>
        <providerOption name="WarnAsError" value="false"/>
      </compiler>
    </compilers>
  </system.codedom>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1" appliesTo="v2.0.50727">
      <dependentAssembly>
        <assemblyIdentity name="System.Web.Extensions" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="3.5.0.0"/>
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Web.Extensions.Design" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="3.5.0.0"/>
      </dependentAssembly>
    </assemblyBinding>
  </runtime>

Removing these made test-explorer able to discover and run my tests.

AbhitejJohn commented 7 years ago

@jehoel: Yes, Replied. @josteink: Thanks. <providerOption name="CompilerVersion" value="v3.5"/> would probably be the culprit. @StingyJack: I suspect this would be a different issue. Could you please raise a new one with details?

MartinF commented 7 years ago

Just want to report that I had the same problems in a newly created project. The app.config used to be empty, but after installing some packages assemblybinding's must have gotten added, which was causing the problem.

The solution was simple - I deleted the whole runtime section and cleared the output folders (bin, obj and TestResults), then it was working again. Tried adding the assemblybindings again and got the same error.

yesitspeter commented 7 years ago

Just wanted to comment that I too am having the same issue with newly created projects. I'm on:

Microsoft Visual Studio Professional 2017 Version 15.3.4 VisualStudio.15.Release/15.3.4+26730.15 Microsoft .NET Framework Version 4.7.02046

Update: I just took that test project, added the .Net standard library through NuGet and it worked. Strange!

StingyJack commented 7 years ago

@StingyJack: I suspect this would be a different issue. Could you please raise a new one with details?

Not without checking these first =D

Some of these are older projects, but not 3.5 old. Something is also broken badly in VS. where running update-package -reinstall on a solution crashes VS halfway through. This leaves us with app.config files for projects displayed in solution explorer as loose files in a solution folder as well as within the projects. Its also forcibly adding assembly binding redirects that are not necessary (internally built dll, only one version ever released even internally). So I think its app.config related for me too, but want to see if fixing them has any effect.

StingyJack commented 7 years ago

I tried downgrading both the test adapter and test framework packages to an older version (.13) and still got test skippage. The only app config content is an assembly binding for a non-test assembly (the one mentioned that shouldnt be needed).

Only removing these two nuget packages entirely and setting the reference mentioned seems to have corrected this. However its only going to work right if everyone uses the same VS install path due to the HintPath it uses.

    <Reference Include="Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>..\..\..\..\..\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll</HintPath>
    </Reference>

Removing the package also doesnt seem to have completely removed the csproj file contents. I removed the following error conditions and import manually to avoid future errors.

  <Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
    <PropertyGroup>
      <ErrorText>This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.  For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
    </PropertyGroup>
    <Error Condition="!Exists('..\..\packages\MSTest.TestAdapter.1.1.18\build\net45\MSTest.TestAdapter.props')" Text="$([System.String]::Format('$(ErrorText)', '..\..\packages\MSTest.TestAdapter.1.1.18\build\net45\MSTest.TestAdapter.props'))" />
    <Error Condition="!Exists('..\..\packages\MSTest.TestAdapter.1.1.18\build\net45\MSTest.TestAdapter.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\..\packages\MSTest.TestAdapter.1.1.18\build\net45\MSTest.TestAdapter.targets'))" />
  </Target>
  <Import Project="..\..\packages\MSTest.TestAdapter.1.1.18\build\net45\MSTest.TestAdapter.targets" Condition="Exists('..\..\packages\MSTest.TestAdapter.1.1.18\build\net45\MSTest.TestAdapter.targets')" />
  <!-- To modify your build process, add your task inside one of the targets below and uncomment it. 
       Other similar extension points exist, see Microsoft.Common.targets.
  <Target Name="BeforeBuild">
  </Target>
  <Target Name="AfterBuild">
  </Target>

@AbhitejJohn - the links on the front page list ways to enable additional diagnostics. Which one should be used for the "Visual Studio Test" build task?

StingyJack commented 7 years ago

The info for enabling diagnostics would be a nice to have at this point. I pulled all the MSTest2 packages and reverted the references, and got all our tests back. Let me know if you need additional info or examples to either reproduce this defect or make sure any proposed correction will work.

AbhitejJohn commented 7 years ago

@StingyJack: For the test task setting System.Debug="true" on the definition or passing in a /diag switch as an additional argument should provide you with more logs. You might need to add a task to upload these logs if they aren't already.

Let me know if you need additional info or examples to either reproduce this defect or make sure any proposed correction will work.

Yes, a repro would help. The proposed correction takes one back to the old MSTest framework which is not Cross-Plat and does not support the new features MSTestV2 has.

pvlakshm commented 7 years ago

@Jehoel,@StingyJack can you share a repro please.

StingyJack commented 7 years ago

I will try to pare down something for this, I can't publicly share the projects that had this problem.

daiplusplus commented 7 years ago

@pvlakshm I don't have it on my computer anymore, but @AbhitejJohn will have it in the email I sent him.

scott-lin commented 7 years ago

I ran into this issue as well today. I was able to workaround / fix my specific issue by removing all unnecessary assembly bindings in app.config. After doing this, my tests were discovered.

How did I determine my necessary bindings? I commented them all out and repeated these steps: (1) run my tests, and (2) reintroduced the binding hinted in the test failure. Luckily, I only had to repeat these steps for two bindings and not for all of my bindings (30+). /phew

Hope this helps.

Davewarner commented 6 years ago

I've got a similar issue I upgraded from 4.6.1 to 4.7.1 of the .net framework now my unit test project can't find and tests, I get the message "given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)" I've tried removing the bindings in the web.config, which did nothing and I've taking a look at some of the logs in fusion but there does not seem to be anything obvious there either. I also tried switching on "setx VS_UTE_Diagnostics 1" but nothing extra came through on the output.

Any suggestions?

StingyJack commented 6 years ago

@AbhitejJohn @pvlakshm

Repro available at https://github.com/StingyJack/testfx241

smadala commented 6 years ago

@StingyJack I couldn't repro the issue. I have tried this on two machines(my dev machine and fresh lab machine with >= VS 15.5 preview 3). testfx-241-repro Can you try with latest VS version? Is this repro with vstest.console.exe(command line) too? If Yes, Can you share logs(output and /diag) of vstest.console.exe?

@Davewarner If you are using VS >= 15.3, You can enable logs by Tools -> Options -> Test -> Logging -> Logging level -> Diagnostic (VS_UTE_Diagnostics option removed in 15.3). And Please provide repro project to try out our side.

StingyJack commented 6 years ago

This has been present for several versions of vs15. Can you please try a non-preview version? Preferably one that has had 15, 15.1., 15.2., etc.

It does work in correctly in both vstest.console.exe and mstest.exe console on my dev box, but this was missing tests on the TFS server too. I'll check that later today.

StingyJack commented 6 years ago

Still misses tests on the TFS build server.

Running this project with the 15.5 preview 5 is more interesting, but also holds the same end result. It compiled the first time, and found only one test in UnitTestProject2. I changed the "keep test execution engine running" to unchecked and compiled and was greeted with errors in the errors list about VS not being able to find the adapter package. Trying to open the nuget package manager ui then pops the warning "No projects supported by nuget in this solution". I reenabled the "keep test exec..." option.

image

Closing the solution and reopening it allows it to build again, and this time it discovers and executes all four expected tests.

Closing all that and reopening the solution in Vs 15 Ent, Vs is immediately able to find the tests and can execute them.

Would the toggling of that "Keep execution engine running" have created some necessary setting?

smadala commented 6 years ago

Closing the solution and reopening it allows it to build again, and this time it discovers and executes all four expected tests.

@StingyJack If I understand correct, test are discovering/executing fine with 15.5. "Keep execution engine running" won't have any effect in 15.5 by default. If you are facing Nuget issue always share repro project and raise issue at https://github.com/NuGet/NuGet.Client.

Still misses tests on the TFS build server.

If locally all tests running fine, then possibly adapters not gettings picked up in TFS, you can determine that by comparing logs files. Collect vstest logs following here from TFS and local vstest logs following here

jemiller0 commented 6 years ago

I was happily executing my tests on VS 2017 with the update prior to 15.5 yesterday. Upgraded to 15.5 late yesterday. Tried running tests today. No tests found and I'm receiving the error about the invalid codebase. How do I downgrade back to 15.4?

jemiller0 commented 6 years ago

Commenting out the bindingRedirects in my test project and then regenerating them seems to have fixed the problem for me. For the time being anyway. The following is what I currently have. The commented out ones are what it sued to be. I used Get-Project -Name OleLibraryTest | Add-BindingRedirect to regenerate the binding redirects.

  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <!--<dependentAssembly>
        <assemblyIdentity name="System.Threading.Tasks.Extensions" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-10.0.0.0" newVersion="10.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Runtime" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Reflection" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Runtime.InteropServices" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Buffers" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.0.2.0" newVersion="4.0.2.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Runtime.Extensions" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Text.RegularExpressions" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.0.0" newVersion="4.1.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.EntityFrameworkCore" publicKeyToken="adb9793829ddae60" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.0.1.0" newVersion="2.0.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.EntityFrameworkCore.Relational" publicKeyToken="adb9793829ddae60" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.0.1.0" newVersion="2.0.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="MySqlConnector" publicKeyToken="d33d3e53aa5f8c92" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-0.33.1.0" newVersion="0.33.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Npgsql" publicKeyToken="5d8b90d52f46fda7" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-3.2.6.0" newVersion="3.2.6.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Net.Http" culture="neutral" publicKeyToken="b03f5f7f11d50a3a" />
        <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.2.0.0" />
      </dependentAssembly>-->
      <dependentAssembly>
        <assemblyIdentity name="System.Threading.Tasks.Extensions" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Buffers" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.0.2.0" newVersion="4.0.2.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.EntityFrameworkCore" publicKeyToken="adb9793829ddae60" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.0.1.0" newVersion="2.0.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.EntityFrameworkCore.Relational" publicKeyToken="adb9793829ddae60" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.0.1.0" newVersion="2.0.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Npgsql" publicKeyToken="5d8b90d52f46fda7" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-3.2.6.0" newVersion="3.2.6.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="MySqlConnector" publicKeyToken="d33d3e53aa5f8c92" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-0.33.1.0" newVersion="0.33.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-10.0.0.0" newVersion="10.0.0.0" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>

Apparently, at least in some cases, the problem appears to have something to do with dependent assemblies.

jemiller0 commented 6 years ago

NuGet in general just seems to introduce DLL hell. I have solved some problems in the past by removing the runtime section in all the App.configs in the projects in my solution and then regenerate them with Get-Project -All | Add-BindingRedirect. It seems to usually remove a lot of unneeded cruft.

smadala commented 6 years ago

@jemiller0 To identify exact assembly which causing the issue by collecting failed assembly bindings logs. And update config file accordingly.

StingyJack commented 6 years ago

If you are facing Nuget issue always share repro project and raise issue at https://github.com/NuGet/NuGet.Client.

@smadala - the reported issue is not a Nuget issue and I'm not sending whole 50-200 project solutions over when they seem to be too busy tinkering with low value things (and breaking stuff along the way) and placation to be bothered with long awaiting bugfixes.

I've only had to log maybe 5 issues in the last 20 years for MSFT. But the amount of time I've had to spend to work around or otherwise deal with recent MSFT quality issues (230+) has totaled about two months for this year. Just this one issue alone has cost me about thee work days, not counting the effect of bugs in software that weren't caught by the unit tests because they were silently skipped during gated checkin builds (they mysteriously stopped working earlier this year - another multi day issue).

If you all could stop screwing around with all the shiny things (nodejs, git, and others your established .net customer base does not care about), and focus on improving the quality of the things that are foundational, I may just get my nights and weekends back in 2018.

Getting additional logs is just more time I lose, and I'm not sure what (if anything) you could gain except more time to service other issues until then. The above information I shared should be enough to repro this. You guys probably need to try it out on a non-MSFT image workstation or other test environments.

smadala commented 6 years ago

@StingyJack Sorry I can't comment on other issues, But I'm very confident on resolving this issue, Help me to find the root cause by providing diagnostics logs. Feel free to send me email if you would like to have skype call.

jemiller0 commented 6 years ago

Would it be possible to improve the error message? It might not be so bad if the error message were better. I was thinking that the problem had something to do with running the tests in 64 bit mode or something. Had I not found this GitHub issue, I would have not thought to check the binding redirects and would have wasted a lot of time. The other thing is, Microsoft really needs to make it possible to go back to older versions of Visual Studio. As far as I know, right now, you can't. You can either go back to the original version of Visual Studio, or the latest update, but not in between updates. And I have got burned a couple times now with issues like this where I install the latest version and then I am dead in the water until I find a fix.

shunty-gh commented 6 years ago

Well I never. Bit late, but I thought I'd better add another report. This issue has been bugging me on and off for a couple of months now. Sometimes the tests are discovered, sometimes not - more often not, though. Yesterday the tests ran to completion, today no tests discovered. No package updates or project changes have happened in the last 24 hours nor any changes to config files. This was using VS 15.5.2. Updated to VS 15.5.3 today - still no tests found. However I can report that removing all the bindingRedirect from app.config has just made all the tests magically reappear. This project was originally .net 4.5 then got upgraded to 4.7. Then the issue started appearing so I deleted the test project and created a brand new one with the right framework level and re-added all the tests. Then it worked... for a little while. But sometimes it would stop working and, more noticeably, fail when a package update happened.

bkwdesign commented 6 years ago

VS 2017 (15.4.1), new .NET Framework 4.6.2 web solution w/test project, + helper library for interpreting OData PATCH deltas. (swapped in OData v4 libraries, added Swashbuckle, Swashbuckle.Odata)

My tests are only discovered after I remove all the bindingRedirect nodes from the assemblyBinding node from App.config.

Am I going to have to add a build step to my CI server to keep this app.config clean? - as I'm certain on a fresh checkout, rebuild, it will become polluted again.

jemiller0 commented 6 years ago

IMHO something is seriously wrong with the code that is used to discover the tests if it is that sensitive to binding redirects. I have no idea why that should have an affect to begin with.

spottedmahn commented 6 years ago

Hi @smadala - please see the attached logs: binding errors.zip

image

StingyJack commented 6 years ago

I'm not sure why nuget has decided its ok to create a slew of app.configs for class libraries, especially when they do not clean them up when they aren't needed later. Its a stupid amount of churn to contend with for code reviews and version control, yet they seem to think its ok.

@smadala - in any of the assembly binding sections posted here does it look like nuget or the default project template is creating or not removing incorrect bindings? I don't have any assembly bindings to offer, as I delete them from class libraries before they get checked in and start the churn.

thiagojsf commented 6 years ago

I had the same issue in a test project at Visual Studio 2017 Professional.

and don't know exactly what solves the problem but it's working now.

I still with an error in the output (Error code 0x80131047). But it does not avoid me to run my tests.

smadala commented 6 years ago

@spottedmahn Is your error code is 0x80070002? Assembly resolving failing for "System.Runtime". C:\Users\mdepouw\Source\Repos\MyProject\MyProject.Library\bin\Debug\MyProject.Library.dll.config has binding redirects for System.Runtime 4.0.0.0 to 4.1.1.0? Look at "System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.html and System.Runtime, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.html".

LOG: Redirect found in application configuration file: 4.0.0.0 redirected to 4.1.1.0.
LOG: Post-policy reference: System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

Remove redirects for System.Runtime or make sure it present in GAC/probing paths.

Can you share steps for how did you get into this problem?

smadala commented 6 years ago

@StingyJack Default template won't create app.config, I'm not sure about nuget or any other tool.

@jemiller0 During tests discovery/run test-adapter(MSTest v2) loads test-assembly(UnitTests.dll) in new appdomain(AD2) with test-assembly config(UnitTests.dll.config) as application configuration file for AD2. In AD2 all the binding redirects(doc) and probing paths(E.g: <probing privatePath="bin;bin2\subbin;bin3"/>) in test-assembly config applied for any assembly loading. That is why assembly load failed if config file has redirects to assembly version which is not present in probing paths. See @spottedmahn shared logs "binding errors\Default\testhost.exe\System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.html" for assembly resolving process.

spottedmahn commented 6 years ago

Hi @smadala - thanks for the quick reply!

No, sorry, that is not my error code. Mine is 0x80131047:

[1/24/2018 8:33:39 AM Informational] ------ Discover test started ------ [1/24/2018 8:33:44 AM Warning] [MSTest][Discovery][C:\Users\mdepouw\Source\Repos\MyProject\MyProject.Library\bin\Debug\MyProject.Library.dll] The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047) [1/24/2018 8:33:44 AM Informational] ========== Discover test finished: 0 found (0:00:05.0329129) ==========

Sorry, I'm not sure the exact steps to repro. I tried to repro it in a sample project but I couldn't.

I was able to figure out some details though. In our SLN, on the failing test project, when I added a Nuget reference to Moq, the app.config file got added. When I tried this in my sample project it did not get added.

I'm guessing the app.config getting added has something to do w/ Automatic Binding Redirection? Thoughts?

Please let me know what other info you need to get this addressed! We can always do a Skype call if that helps, just let me know.

smadala commented 6 years ago

@spottedmahn Sent you mail for meeting.

spottedmahn commented 6 years ago

Hi @smadala - I've managed to reproduce it in a sample app 🎉

Please see this project and readme.md for details.

StingyJack commented 6 years ago

@smadala - Since you can run our repro cases without issue, this is probably environmental. Per @spottedmahn 's fusion log dump, this is the config file used in the reports of an assembly load failure. Does this testhost.exe.config file look correct? 

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <startup useLegacyV2RuntimeActivationPolicy="true">
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0" />
  </startup>
  <runtime>
    <legacyUnhandledExceptionPolicy enabled="1" />
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <probing privatePath="Extensions" />
      <!-- Test adapters compiled against version 11-14, need to be redirected to version 15. -->
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.TestPlatform.ObjectModel" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="11.0.0.0-14.0.0.0" newVersion="15.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.TestWindow.Interfaces" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="11.0.0.0-14.0.0.0" newVersion="15.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.UnitTestFramework" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="10.1.0.0" newVersion="10.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.Resource" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="15.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.ControllerObject" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="15.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.ExecutionCommon" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="15.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.Common" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="15.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="10.0.0.0-14.0.0.0" newVersion="15.0.0.0" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>
  <system.diagnostics>
    <switches>
      <add name="TpTraceLevel" value="0" />
    </switches>
  </system.diagnostics>
  <appSettings>
    <!--<add key="ExecutionThreadApartmentState" value ="MTA"/>-->
    <!--<add key="TraceLogMaxFileSizeInKb" value ="10240"/>-->
    <!-- MsTest Adapter Specific AppSettings -->

    <!-- This flag is added to support test execution for net35 tests through TMI adapter. -->
    <add key="TestProjectRetargetTo35Allowed" value="true" />
  </appSettings>
</configuration>

I have slightly later versions on the files in that folder (not sure if windows shows the asm ver) image

spottedmahn commented 6 years ago

@smadala - Since you can run our repro cases without issue

Hi @StingyJack - have you tried these repro steps? I'm pretty sure @smadala should be able to reproduce the issue now 🎉

StingyJack commented 6 years ago

@spottedmahn - i think so, except I dont have any other referenced packages.

spottedmahn commented 6 years ago

Hi @StingyJack - oh, sorry, didn't see your repro! 👍