Generated test cases may not appear in Test Explorer from .NET Framework compiled test projects on VS2015 with a custom value provider

[mwpowellhtx]

After a round of discussions with my ReSharper team, they discovered that applications created under .NET Core tested fine. However, applications created under .NET Framework did not. See the xunit issue. I suspect that should that get resolved, the issue should take care of itself. My suggestion would be to provide .NET Framework support in addition to .NET Core support. I don't know exactly how that's accomplished, via NuGet dependencies, etc, perhaps.

[AArnott]

Are you under the impression that this project does not work when the tests run on. NET Framework? I personally use this project on both runtimes. What problem are you having on the .NET Framework?

[mwpowellhtx]

I explained why it didn't work. The combinatorial combinations were failing to discover under the VS test runner, as evidenced by the exception thrown.

On Mon, May 7, 2018 at 11:38 PM, Andrew Arnott notifications@github.com wrote:

Are you under the impression that this project does not work when the tests run on. NET Framework? I personally use this project on both runtimes. What problem are you having on the .NET Framework?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/AArnott/Xunit.Combinatorial/issues/17#issuecomment-387274709, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvVu7-WAXrHScgt6pVkYbRhy8k9efwsks5twRMrgaJpZM4T133G .

[AArnott]

Thanks. Nowhere in your original issue did you mention the VS test runner or discovery. I'll see if I can repro your problem and get back to you. It certainly isn't our intent to "exclusively leap into .NET Core".

[mwpowellhtx]

I apologize if I wasn't clear there. As far as I've been told, it's a difference between VS2017 discovery versus VS2015 non-discovery (i.e. exception). I'm still running against Framework, desktop, and giving Core a chance to mature a bit before I make any serious commitments. Not least of which is a lack of WinForms, WPF, etc, support.

On Tue, May 8, 2018 at 1:12 AM, Andrew Arnott notifications@github.com wrote:

Thanks. Nowhere in your original issue did you mention the VS test runner or discovery. I'll see if I can repro your problem and get back to you. It certainly isn't our intent to "exclusively leap into .NET Core".

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/AArnott/Xunit.Combinatorial/issues/17#issuecomment-387286443, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvVu9eYglzD4FsWf9cGp6ielIdtUQJfks5twSk0gaJpZM4T133G .

[AArnott]

This is what I see in VS 2017. It seems to discover all the generated test cases just fine. The test project is a multi-targeting one, but as net45 is the first in the list, I believe the Test Explorer is running discover on and executing tests from that framework. So it looks like it's working. Is that inconsistent with what you see?

[mwpowellhtx]

I was getting exceptions during test discovery. I am using a custom combinatorial values attribute.

[AArnott]

In that case, I would suspect your custom values attribute may be what only works on .NET Core. I haven't seen your code so I can't say how likely that is.

[mwpowellhtx]

I assure you it is not "my code" in this case. I don't do anything all that fancy. The key is that I want to statically initialize my values for performance reasons, especially in cases where the test method, theory, has a breadth of possible parametric combinations. Otherwise testing just gets bogged down in discovery at any given phase. The exceptions I received (see previous cases) led me to the xunit stack.

On Tue, May 8, 2018 at 7:08 PM, Andrew Arnott notifications@github.com wrote:

In that case, I would suspect your custom values attribute may be what only works on .NET Core. I haven't seen your code so I can't say how likely that is.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/AArnott/Xunit.Combinatorial/issues/17#issuecomment-387570274, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvVuxCYBymOC6kbgnUsSqkuvy7EQAubks5twiVWgaJpZM4T133G .

[mwpowellhtx]

To be clear, it does not work under VS2015. That's my priority. I don't care about .NET "Core", at least not until it is more proven. Never mind there is no viable desktop migration path for it today. But that's neither here nor there re: xunit, combinatorial, etc. Point is, epic fail under VS2015 test discovery.

On Wed, May 9, 2018 at 8:47 AM, Michael Powell mwpowellhtx@gmail.com wrote:

I assure you it is not "my code" in this case. I don't do anything all that fancy. The key is that I want to statically initialize my values for performance reasons, especially in cases where the test method, theory, has a breadth of possible parametric combinations. Otherwise testing just gets bogged down in discovery at any given phase. The exceptions I received (see previous cases) led me to the xunit stack.

On Tue, May 8, 2018 at 7:08 PM, Andrew Arnott notifications@github.com wrote:

In that case, I would suspect your custom values attribute may be what only works on .NET Core. I haven't seen your code so I can't say how likely that is.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/AArnott/Xunit.Combinatorial/issues/17#issuecomment-387570274, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvVuxCYBymOC6kbgnUsSqkuvy7EQAubks5twiVWgaJpZM4T133G .

[AArnott]

Ah, VS2015 is also a very useful detail to know.

Can you please provide a sample that repros the problem?

[mwpowellhtx]

Sure:

class MyValuesAttribute : CombinatorialValuesAttribute {

private static IEnumerable GetValues() { yield return 0; yield return 1; yield return 2; }

private static object[] Values { get; } = GetValues().ToArray

[AArnott]

It works great for me. I'm in VS 2015, in a net452 target framework class library, and all the tests show up in Test Explorer:

I'm going to close this issue as Not Repro. If you care to share a URL to a full repo or send a zip with a fully working, compiling and reproing case, I can take another look.

[mwpowellhtx]

Hmm, interesting... That's in the test explorer. Did you try in the actual test runner? Starting to sound like perhaps an extension issue more than a test framework issue, per se, but I just wanted to make sure.

On Wed, May 9, 2018 at 6:29 PM, Andrew Arnott notifications@github.com wrote:

It works great for me. I'm in VS 2015, in a net452 target framework class library, and all the tests show up in Test Explorer:

[image: image] https://user-images.githubusercontent.com/3548/39842929-846a3e66-539d-11e8-97ef-60fd4f7b886f.png

I'm going to close this issue as Not Repro. If you care to share a URL to a full repo or send a zip with a fully working, compiling and reproing case, I can take another look.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/AArnott/Xunit.Combinatorial/issues/17#issuecomment-387894634, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvVu2_MdEhP3LV3H9cGaD-q6Bfm-083ks5tw23FgaJpZM4T133G .

[AArnott]

What test runner? The Test Explorer can run the tests. Are you talking about the console runner? (vstest.console.exe)

[mwpowellhtx]

Ah, I see... I think it's a R# now that I look at it again. The errors being what they are, if I just follow the plain error, it is obviously JetBrains related.

2018.05.09 19:43:54.574 ERROR JetBrains.Util.InternalErrorException: Unexpected: Unable to find any matching test cases at JetBrains.ReSharper.UnitTestRunner.Xunit.TestRunner.Run(XunitTestAssemblyTaskassemblyTask) The exception happens even with nominal, i.e. non-derived, CombinatorialValuesAttribute usage.
2018.05.09 19:43:54.626 WARN Element <Xunit.Verification.MyTests> was left pending after its run completion.
2018.05.09 19:43:54.631 WARN Element <Xunit.Verification.MyTests.Verify> was left pending after its run completion.

[mwpowellhtx]

Verification.zip

[AArnott]

You're missing the xunit.runner.visualstudio package. I installed that.

I see your repro targets net45, whereas mine targeted net452. xunit installed, bringing in netstandard1.1 dependencies instead of its net452 dependencies, and as a result, it's prepping for a dotnet test run instead of a .NET Framework test run. I retargeted to net452, then ran the Update-Package -Reinstall command in the Package Manager Console.

With those changes, it works in Test Explorer:

So the crux of it for Test Explorer is that you need to target net452 at least, since that's what xunit supports. That doesn't have to impact your product code -- it's just your test assembly that needs to target at least that new of the .NET Framework. And for Test Explorer you need the xunit.runner.visualstudio package installed.

I don't use Resharper.

[mwpowellhtx]

I updated the framework target, and still finding the same error. I think at this point it is simply that: R#.

On Wed, May 9, 2018 at 8:36 PM, Andrew Arnott notifications@github.com wrote:

You're missing the xunit.runner.visualstudio package. I installed that.

I see your repro targets net45, whereas mine targeted net452. xunit installed, bringing in netstandard1.1 dependencies instead of its net452 dependencies, and as a result, it's prepping for a dotnet test run instead of a .NET Framework test run. I retargeted to net452, then ran the Update-Package -Reinstall command in the Package Manager Console.

With those changes, it works in Test Explorer:

[image: image] https://user-images.githubusercontent.com/3548/39846229-5e3d5cb6-53af-11e8-8147-1fb3e9d45e28.png

So the crux of it for Test Explorer is that you need to target net452 at least, since that's what xunit supports. That doesn't have to impact your product code -- it's just your test assembly that needs to target at least that new of the .NET Framework. And for Test Explorer you need the xunit.runner.visualstudio package installed.

I don't use Resharper.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/AArnott/Xunit.Combinatorial/issues/17#issuecomment-387915696, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvVu_OAALCmzxXxlmA9R3gwEUQlE0liks5tw4t-gaJpZM4T133G .