System.InvalidOperationException and System.Collections.Generic.KeyNotFoundException errors when running tests

[liamfl]

Description

I am running unit tests on a 2 c# test projects using the lates version of Visual Studio Community 2022

The 2 test project are named EngineTests and ServerCodeTests.

While the test projects compile correctly, I regularly get a series of error messages in the Output / Tests window when running unit tests:

System.InvalidOperationException: TestsForOrigin: Unknown TestCaseRecord.Origin value Unknown System.Collections.Generic.KeyNotFoundException: The given key 'C:\Users\liamf\Source\Repos\TradingReplay\ServerCodeTests\bin\Debug\net6.0\ServerCodeTests.dll' was not present in the dictionary. An exception occurred while test discoverer 'MSTestDiscoverer' was loading tests. Exception: Could not load type 'Microsoft.VisualStudio.TestTools.UnitTesting.StringEx' from assembly 'Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. No test is available in C:\Users\liamf\Source\Repos\TradingReplay\ServerCodeTests\bin\Debug\net6.0\ServerCodeTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Steps to reproduce

Running any test from Test Explorer.

Reloading Visual Studio allows you to do 2-3 test runs before the issue arises again.

Diagnostic logs

ErrorLog.text attached

Environment

Windows 11 10.0.22621 Build 22621 Visual Studiio Community v17.4.3

Nuget packages: MSTest.TesAdapter v3 MSTest.TestFramework v3 Microsoft.NET.Test.SDK v17.4 ErrorLog.txt

[peterwald]

This is not fixed in Visual Studio 2022 Enterprise V17.5.4

I am using NUNIT, not MSTest

@TomKuhn, the original issue was due to an interaction between MSTest and the Visual Studio Test Explorer. You must be seeing a different condition that generates a similar error.

Can you help us understand how to reproduce the error with NUnit?

[TomKuhn]

@peterwald It is very similar, in that seemingly randomly, it will fail to execute any tests with the test explorer displaying many errors (exceptions) matching the above post. Only a restart of visual studio will fix it. Sadly I have had to restart VS about 30 times today :( It does not seem to have anything to do with renaming/removing tests, but repeatedly debugging a test and changing source code while debugging seems to make it happen.

Versions:

Visual Studio: 17.5.4 Nunit nuget package: 3.13.3 Nunit3TestAdapter: 4.4.2 .Net Framework 4.7.2

Old-style project format: <?xml version="1.0" encoding="utf-8"?>

... ... If there's anything else I can provide, I'm happy to do so Tom

[TomKuhn]

We do use [TestCaseSource(nameof(LanguagesTestData))] a lot which allows us to dynamically create tests based on executing code, which is an Nunit feature that's quite scary!

[peterwald]

We do use [TestCaseSource(nameof(LanguagesTestData))] a lot which allows us to dynamically create tests based on executing code, which is an Nunit feature that's quite scary!

I've been unable to reproduce the problem using data-driven (TestCaseSource) NUnit tests. Is it possible for you to share a skeleton of what your tests look like? Are you using generics in your tests? Unusual or custom test names? What types are the TestCaseSources providing? Any other tips on what operations trigger the error?

[abatishchev]

Shall the issue be split? The original once was about MSTest, the new one is about NUnit, even if similar - it would benefit everyone to track it separately. Agreed?

[Evangelink]

@abatishchev @TomKuhn I agree it would be better to open a separate issue. Test Explorer related issues (TestStore entry in the exception) are usually better tracked at https://developercommunity.visualstudio.com/home but if it's more convenient for you please do open another issue here and we will tag it with IDE.

[phuong-leadsoft]

This issue still here on VS 2022 17.6.2

[nohwnd]

please file that as a feedback to VS, using this little fellow on top of you VS:

[IanKemp]

I'm seeing these exceptions too in 17.6.3, using XUnit. And I haven't renamed anything; literally just opened VS and tried to run tests.

This is a very annoying and disappointing flaw in what is supposed to be a developer productivity product; I've been resisting the switch to Rider that many of my teammates have made, but the fact that JetBrains are at least able to ensure that core functionality doesn't break for no good reason... is becoming more and more difficult to ignore.

[Evangelink]

I have opened the following issue on dev community for you to track and follow - please feel free to add comment to it: https://developercommunity.visualstudio.com/t/SystemInvalidOperationException-and-Sys/10411793.

Closing this ticket as issue is not on Test Platform side.

[olivier-spinelli]

Same issue using (latest version): Microsoft.NET.Test.Sdk 17.6.3 NUnit3TestAdapter 4.5.0 NUnit 3.13.3

https://developercommunity.visualstudio.com/t/SystemInvalidOperationException-and-Sys/10411793 is closed. Any change in the context breaks all. The only workaround I found is to:

• close VS
• delete the folder: .vs\\v17\TestStore.
• restart VS Painful (to say the least). Didn't find another issue about this. Should I open a new one?

[Evangelink]

@olivier-spinelli the issue is marked as "under investigation" on dev community website (not as closed).

Ping @AbhitejJohn @peterwald

[olivier-spinelli]

Glad to hear that... But why, being logged in, can't I see it? (Subscription "Visual Studio Enterprise (MPN)")

Viewing as ol***li@signature.one We were unable to get this feedback item. It could be because you don't have access to it or it doesn't exist

Error details: ["Request failed with status code 403"]

[Evangelink]

Oh thanks for the info! I logged in with my personal account and get the same error as you but I can see it correctly with work account. I will check with colleagues what's going-on and will get back to you!

[Evangelink]

@olivier-spinelli the issue is now public! I didn't pay attention that I created it as internal.

[olivier-spinelli]

Many thanks!