Test Explorer fails to show tests. Tests output: "The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)"

[daiplusplus]

Description

I cloned a private Git repo on my local machine. The solution comprises of several projects all targeting .NET 4.6.2. I built the solution which pulled-in some NuGet packages, including MSTest.TestAdapter, 1.1.18 and MSTest.TestFramework, 1.1.18. There is a test project in the solution with several [TestClass]/[TestMethod] tests. The solution builds successfully, however the Test Explorer window is empty. On a coworker's machine running VS2015 the Test Explorer is populated correctly.

In the Output window's Test output I see the following:

------ Discover test started ------ The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047) ========== Discover test finished: 0 found (0:00:00.1886778) ==========

Steps to reproduce

I have no specific reproduction steps - it happened spontaneously on my machine. Restarting VS, cleaning the project output and TestResults directory, and closing+reopening the Test Explorer window has no effect.

Expected behavior

The tests in the project should appear in Test Explorer.

Actual behavior

No tests appear in Test Explorer, and an error message in the Output window.

Environment

Please share additional details about the test environment. Operating system, Build version of vstest.console, Package version of MSTest framework and adapter

• Windows 10 x64 Enterprise, Creators Update
• Visual Studio 2017 Enterprise, 15.2
• MSTest package version 1.1.18
• Projects targeting .NET 4.6.2
• I have two copies of vstest.console installed:
  • C:\Program Files (x86)\Microsoft Visual Studio 15.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe - 15.0.26228.0 - 142KB
  • C:\Program Files (x86)\Microsoft Visual Studio 15.0\Common7\IDE\Extensions\TestPlatform\vstest.console.exe - 15.0.0.0 - 117KB

[StingyJack]

@spottedmahn - long thread is long

[smadala]

@spottedmahn Thanks for repro. It seems like core issue (Fail to load assembly(System.Runtime)) not related to testadapter.

Create console app similar to test project you created and add Assembly.Load("System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"); causes same failure as test discovery/run. Copying System.Runtime.dll from ClassLibrary's OutputDirectory to ClassLibrary.Tests/ConsoleApp OutputDirectory fixes the assembly load.

In meeting, We thought System.Runtime.GCSettings.IsServerGC from System.Runtime assembly, Actually it is from mscorlib.dll.

I'm not sure whose(NuGet.Client/ msbuild) issue is this? Can you create issue on them?

From MSTest testadapter side, Error message need to improve. I will keep this issue open to address it.

[StingyJack]

From MSTest testadapter side, Error message need to improve. I will keep this issue open to address it.

@smadala - there is also the issue of it continuing to execute and "skip" tests without any failure indication or exception. It did this for months on the build servers before I noticed it happening locally

[smadala]

@StingyJack Thanks for bring that up, As this thread getting too long, We will track the issue separately. Please create another issue with repro steps and logs(/diag:log.txt).

[StingyJack]

Per the title, it is the same issue. the thread is only getting too long as its not been resolved and more people are submitting claims and evidence.

[smadala]

@StingyJack Can you share repro and logs again?

[spottedmahn]

Hi @smadala - can you publish your sample app to GitHub?

Create console app similar to test project you created and add Assembly.Load("System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"); causes same failure as test discovery/run. Copying System.Runtime.dll from ClassLibrary's OutputDirectory to ClassLibrary.Tests/ConsoleApp OutputDirectory fixes the assembly load.

[StingyJack]

@smadala - here and here

When you ran the same test project using (close to) the same tools, it works for you. While it could be a small difference in code causing it, this scenario means its probably environmental - a config file, operating system, work process, etc. The config file looks like a good place to start, and @spottedmahn has been kind enough to share fusion logs that show a failure to load which matches the title of this issue.

What did you find after looking into that?

[spottedmahn]

Hi @smadala - I'm unable to reproduce it with a console app. Please see this code. Can you share your repro?

static void Main(string[] args)
{
    Assembly.Load("System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a");
}

[smadala]

@spottedmahn Apologies for delay in response, Here is the repro. Looking at your console app, It doesn't have reference to Moq (no app.config).

[spottedmahn]

Hi @smadala - when you run that sample it fails but it does have an app.config:

If comment out the redirect it doesn't fail.

---

Again, thanks for your help on this. Still not sure who to report this issue to...

[smadala]

Start at nuget, Because they are creating redirects but not target/something to copy the expected assembly to bin directory.

[StingyJack]

@smadala - nuget team has rejected this issue due to insufficient information. I objected, but it remains closed.

[spottedmahn]

I submitted an issue here on Developer Community, hopefully, it will get some traction.

[ManishJayaswal]

Folks - lets follow this issue on the developer community. This thread has become too unwieldy.

[StingyJack]

Per the title, it is the same issue. the thread is only getting too long as its not been resolved and more people are submitting claims and evidence.

Its not too unweildy for me, and the dev comm site is frustrating in so many ways (cant search through own issues, formatting is ne'er impossible, notifications, mods can lock out OP replies, etc), why prefer it?

[jemiller0]

Microsoft should just get rid of their forum websites and use Stackoverflow completely. Every time I have a problem and I do a web search and end up at Microsoft's site, I find completely useless information. Usually, it is from a Microsoft employee who clearly is a newbie who doesn't have a clue. It is just a waste of time even looking at that site.

[ManishJayaswal]

I should have provided a closure on this issue here as well so sorry about that.

This is with reference to the repro provided by @spottedmahn at (https://github.com/spottedmahn/Experiments/tree/master/Test%20Explorer%20fails%20to%20show%20tests ) I noticed that certain references were not resolved by NuGet after solution load. I found this by expanding the references node on the project where the Test Explorer window was not able to find the tests. They were showing the yellow triangle against the test framework assemblies. Once I did a build, the references were resolved by NuGet and all 4 tests were discovered. NuGet not resolving all references immediately on solution load is a known issue that the NuGet and project system team is aware of. The workaround is to build the solution. I was using 15.6 preview build. Please note that we have made several improvements to TE window in this release which included speeding up performance, adding a new hierarchy view and real time test discovery which discovers tests without requiring a build. Please give it a try and let us know what you think. We expect to release this soon.

[spottedmahn]

Hi @ManishJayaswal - thanks for the update.

I don't have any unresolved references and I get this error:

[spottedmahn]

Hi @ManishJayaswal - also, I think you're using a different repro. The link you provided only has 2 projects.

vs

[ManishJayaswal]

@spottedmahn - can you please point me to the right repro case. I was using the one provided in the developer community ticket that you had opened couple of days ago.

[spottedmahn]

Hi @ManishJayaswal - the link you provided is it but it does not match your screenshot.

[ManishJayaswal]

Ah sorry - I was using this repro provided by @StingyJack - (https://github.com/StingyJack/testfx241). We will try the other one too and post back our findings. @spottedmahn - Thanks for pointing this out!

[spottedmahn]

great, thanks @ManishJayaswal for digging in. Please let me know if I can help at all to get to the bottom of this!

FYI, if you comment out the binding redirect in app.config for "System.Runtime" it will discover the tests.

[ManishJayaswal]

@spottedmahn I tried your repro on the latest 15.6 preview bits ( which just got released - like 15 minutes ago- so you can download and try it too). When I opened the solution, I noticed that the reference to System.Runtime ( blue circle in the image below in solution explorer window - kind of hard to see, sorry about my poor choice of color) was not resolved ( had a yellow triangle on it). However it got resolved when I built the solution and then the tests were getting discovered. Real time test discovery was also working ( I verified it by adding a removing a new test and TE window was getting update in real time). Please give the latest preview a try and let us know if this is still an issue.

[spottedmahn]

So the tests are discovered by but not runnable. Did you try running that test?

Test Output Window

[2/23/2018 2:25:01 PM Informational] ------ Load Playlist started ------ [2/23/2018 2:25:01 PM Informational] ========== Load Playlist finished (0:00:00.0025067) ========== [2/23/2018 2:25:27 PM Informational] ------ Discover test started ------ [2/23/2018 2:25:31 PM Warning] [MSTest][Discovery][C:\Users\mdepouw\source\repos\GitHub\spottedmahn\Experiments\Test Explorer fails to show tests\ClassLibrary.Tests\bin\Debug\ClassLibrary.Tests.dll] The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047) [2/23/2018 2:25:31 PM Informational] ========== Discover test finished: 0 found (0:00:04.0140926) ========== [2/23/2018 2:26:31 PM Informational] ------ Run test started ------ [2/23/2018 2:26:32 PM Warning] [MSTest][Discovery][C:\Users\mdepouw\source\repos\GitHub\spottedmahn\Experiments\Test Explorer fails to show tests\ClassLibrary.Tests\bin\Debug\ClassLibrary.Tests.dll] The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047) [2/23/2018 2:26:32 PM Warning] No test is available in C:\Users\mdepouw\source\repos\GitHub\spottedmahn\Experiments\Test Explorer fails to show tests\ClassLibrary.Tests\bin\Debug\ClassLibrary.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again. [2/23/2018 2:26:32 PM Informational] ========== Run test finished: 0 run (0:00:01.1369673) ==========

@ManishJayaswal

[ManishJayaswal]

@spottedmahn - you are right the execution does not work and removing the binding redirect makes the test run. This indicates that TestPlatform has a dependency on a newer version of System.Runtime. I will follow up with TestPlatform team to understand this better but can you tell us why you need to depend on an earlier version of System.Runtime?

@singhsarab FYI

[spottedmahn]

Hi @ManishJayaswal - I'm glad you are able to repro it. Please let us know what you find out.

can you tell us why you need to depend on an earlier version of System.Runtime?

I don't know. I just added some Nuget packages and that file got generated. I'm guessing it has something to do w/ Automatic Binding Redirection

[ManishJayaswal]

As a workaround for now, you can comment out the binding redirect ( which I believe you are already doing). Test will use newer System.Runtime and it should be fine. We would need to investigate to see if we can move TestPlatform dependency to a lower version but I don't expect that change to happen quickly. Thanks for helping us get to the root of this issue. Much appreciated @spottedmahn !!

[StingyJack]

from https://developercommunity.visualstudio.com/comments/202322/view.html

I have responded on the other GH issue. that particular issue also seems to be fixed in the latest 15.6 previews. -Manish Jayaswal

The "other GH issue" is this one

from right above this comment...

We would need to investigate to see if we can move TestPlatform dependency to a lower version but I don't expect that change to happen quickly. -Manish Jayaswal

@ManishJayaswal - Is it fixed or not? On the same day you said it was fixed and then not.

[spottedmahn]

Hi @StingyJack - I think @ManishJayaswal was thinking it was fixed b/c the test discover does work in 15.6 but the tests fail to run now. So 15.6 has just moved issue to a different place.

[StingyJack]

While that intention is possible, after now having multiple encounters the pattern tends to be about improving issue closure stats and not issue resolution. Ofc, YMMV.

[ManishJayaswal]

@StingyJack - I can categorically state that closure stats is not even a consideration in the company. I am mentioning this since you have brought this up several times.

In this specific case - as I have mentioned to @spottedmahn above, please try the simple workaround of removing the binding redirect. TestPlatform ideally should take dependency on the minimum possible version of System.Runtime assembly so we are working with the TestPlatform team to see what can be done here. Hence we are keeping the issue open. However please let us know if the workaround does not work for you and if this issue is a blocking issue for you.

[StingyJack]

As stated on both Skype and here, I can revert to the original QualityTest dll's and the build server no longer silently fails to discover/execute tests. This is the only usable workaround as Nuget wont stop "fixing" binding redirects by "helping" me and putting binding redirects in that do not work or are not necessary. In the case of a test project and other dll projects, this means I have to keep undoing the addition of the app.config - a file I cant really .tfignore without being overly specific. And this specificity earns me yet another thing to maintain.

Regarding the pattern, this is what it looks like from this side.

Report for open items executed on 02/26/2018 09:54:00
There are 269 items in 'My Followed Items' list
There are 215 items I've authored of 269 total followed
There are 2 items that need an update from me
There are 2 items that need tagging by me
There are 137 items that need an update from MSFT
There are 12 items that are still Triaged (or [PERMA TRIAGE])
There are 128 items that need no update
Items I consider closed 50
Items MSFT considers closed 109
Items I dont feel like arguing about 16

And none of that explains the difference in the two statements made in the same day.

[spottedmahn]

This is the only usable workaround as Nuget wont stop "fixing" binding redirects by "helping" me and putting binding redirects in that do not work or are not necessary

I wonder if this is Nuget or Visual Studio? My understanding is it is Visual Studio.

Starting with Visual Studio 2013, when you compile apps that target the .NET Framework 4.5.1, binding redirects may be automatically added to the app configuration file to override assembly unification.

Source: Automatic Binding Redirection

[StingyJack]

I'm not sure I care if its nuget or VS really. Its bugs are creating more work for me in addition to creating this nasty bug. I just want them fixed without catching a bunch of flak.

[cltshivash]

The issue is with the auto redirection being added without the right assemblies being present. More details below.

1: Repro :(https://github.com/StingyJack/testfx241)

This works fine.

2: Repro (https://github.com/smadala/samples/tree/master/Test%20Explorer%20fails%20to%20show%20tests/ClassLibrary.Tests) https://github.com/spottedmahn/Experiments/tree/master/Test%20Explorer%20fails%20to%20show%20tests

Able to repro with the both projects.

-> Tests are being discovered fine through the source based discovery -> Test Execution fails.

Detailed Repro Steps

1. There’s no binding redirect added on adding just the system.runtime’s 4.3.0 version to the classlibrary project and creating a test project exercising the classlibrary.

2. Execute the test, the tests execute fine with System.RunTime.dll being picked from GAC (4.0.0.0) version (which is the test platform/framework dependency)

3. When Moq.dll is added to the test project, the app.config of the test project gets updated with the redirect for the System.RunTime to 4.1.1.0. There’s no reference added to the test project appropriate dll (System.RunTime.dll 4.1.1.0 version).

4. When the tests execute the configuration provided in dll.config is used to setup an appdomain for test execution. The appbase for the appdomain is the output directory of the test project. Since the System.Runtime.dll (4.1.1.0) is not present in the test project output path the execution fails (the error message doesn't indicate the dll that failed and has been fixed)

5. Now uninstall Moq.dll (which added the redirection in the first place). The redirection still stays in the test project.

6. The issue doesn’t repro when the test projects target 4.7.1 (looks like some consolidation that’s being talked of here https://github.com/dotnet/corefx/issues/25773 is helping)

The issue can be mitigated by either removing the redirects or by adding a reference to the 4.1.1.0 version of the System.Runtime.dll and ensuring that it is present in the test output path.

[StingyJack]

Finally, some truth... @cltshivash - where have you been this whole time? =)

Please make sure this fix works for tfs 2015.4 vnext test task. That was where it started silently failing for my team.

[cltshivash]

@StingyJack The fix isn't at the task end so it should work. Nevertheless we will validate once we have a resolution.

[smadala]

Update your MSTest.TestAdapter to 1.2.1 and MSTest.TestFramework to 1.2.1.

These packages will be available in nuget.org very soon. Until then you can obtain them from feed https://dotnet.myget.org/F/mstestv2/api/v3/index.json

After updating you can see message with exact missing assembly name.

I'm closing this issue as there is no more action required from testfx side.

[StingyJack]

After updating you can see message with exact missing assembly name.

@smadala - getting the assembly name is only identifying the assembly that is missing. The STR and E/A complain about tests not being discovered, not that we don't see the assembly name its complaining about.

I'm closing this issue as there is no more action required from testfx side.

If the assembly that is missing (I presume you know what that assembly is at this point) is a MSFT assembly, then there is still action required from the MSFT side.

As someone who has too many years in the same kind of customer service role you are holding in within this thread, I offer you the advice of not prematurely closing this thread. When the package is available and the OP or participants in this thread deem it fixed satisfactorily, that would be the time to close it. @ManishJayaswal attests that there are no issue closure stats being tracked at MSFT for measuring individual performance, so leaving this open until that agreement happens should not pose much of a problem, if any.

[spottedmahn]

I offer you the advice of not prematurely closing this thread. When the package is available and the OP or participants in this thread deem it fixed satisfactorily, that would be the time to close it.

+1

Well put @StingyJack ⚡!

[cltshivash]

Please check if migrating the nuget packages from package.config to package references helps.

For more details please refer to the issue on the nuget repo. NuGet/Home#6723

[StingyJack]

@cltshivash - The nuget team owes a conversion/migration tool (its close to done I think) before we can realistically convert to package ref. If the problem with the silent failure to discover tests gets fixed I can at least not worry about that part of it. I'd rather have it break on build so we can workaround it sooner.

[cltshivash]

@StingyJack Does the option available in 15.7.0 help with the migration ?

[StingyJack]

@cltshivash - A tool is for sure better than the semi-manual steps. I'd still have to click on it for every single project (10 - 180 per solution) unless I can action it at the solution level.

However, the first three of these, in addition to the "Not yet available for Asp.net full framework" are complete blockers for us. Thanks for the info share though.

[cltshivash]

@mishra14 Do you have recommendation for the below ?

• Ability to migrate at solution level
• Timeline on when would Package reference be available for Asp.net full framework

[mishra14]

@cltshivash

1. We have considered a solution level migration. We plan on supporting it in future. You can track it at https://github.com/NuGet/Home/issues/5879.
2. We have a plan of supporting PR on ASP.NET full framework, but there are some open questions that are still need to be sketched out. You can follow the larger effort to improve the PC to PR experience at https://github.com/NuGet/Home/issues/5877.

[spottedmahn]

Please check if migrating the nuget packages from package.config to package references helps.

Yes, that worked on my repro 😊. Branch w/ conversion to Package Reference.

[Evangelink]

@Haplois Is there any work we need to do for this issue?