Add ReSharper / Visual Studio test runner

[Aaronontheweb]

It would be really nice to be able to debug these NBench tests inside visual studio!

[richardszalay]

Adding to this, xunit manages this with a NuGet-based package, which is certainly preferable to a VS extension.

[Aaronontheweb]

@richardszalay that's great to know - much easier for us to CI and deploy that way too

[angellaa]

I would still love to have integration with ReSharper

[mcquiggd]

@angellaa

Andrea, based on your code for NUnit, I have a solution for running NBench benchmarks and reporting results using XUnit 2 and ReSharper.

Have you come up with something in the meantime?

I wrote this quickly last night, and will place on Github.

@RehanSaeed - you were looking for something like this too, I believe?

[RehanSaeed]

@mcquiggd Yes, sounds great!

[angellaa]

@mcquiggd Sound great. No unfortunately I didn't have time to play with NBench since then.

[mcquiggd]

Hi @angellaa, @RehanSaeed

I put the initial attempt up here:

https://github.com/Pro-Coded/Pro.NBench.xUnit

It's was a little tricky to get output of measurement data at first, as NBench requires a parameterless constructor, while xUnit 2.x requires a ITestOutputHelper to be passed into a constructor to allow output; it also needs an xUnit 'test context' to write the output.

So inheritance from a base class was out - hence the current arrangement. It works and is not too messy. I basically needed this for other projects I am building.

As mentioned in the readme, if I have time I will look at a custom FactAttribute / TheoryAttribute / Discoverer. That would allow 'pre-discovery' of NBench tests, more seamless integration, and potentially simplify things slightly.

My own requirements are potentially outputting the standard markdown reports available from NBench, and also collating measurements from multiple tests for direct comparisons.

Let me know your thoughts about what it currently there, and I will look at making it a NuGet package.

David

[angellaa]

Thanks for mentioning me in the project. I am glad my blog posts has being useful. I don't have time to actually play the code right now but it looks good to me. Having these extensions as NuGet packages would make it a lot easier to use so go ahead.... well done! we should definitely create something similar for NUnit

[mcquiggd]

As an update, I also spent some time yesterday looking at creating an xUnit Fact Attribute for NBench; this works in that all Tests annotated with the custom NBenchFact are discovered, and can be executed, with metrics and assertions being correctly determined.

The same problem exists as before however (and which is proving problematic for several other projects that people wish to integrate with xUnit) - getting the output displayed in the Test Runner.

From xUnit 2.x onwards, the ITestOutputHelper has to be injected via the constructor, and the constructor is called for every Test. You cannot 'new up' a TestOutputHelper as it is scoped to the current Test, and can only be supplied and listened to by the xUnit framework.

Subsequently, NBench also creates it's own instance of the Test class using Activator.CreateInstance, and requires a parameterless constructor. I've looked at passing the class instance instantiated (and hooked by XUnit) down into NBench, but that would require some changes to both projects which are not really practical.

The NBench source includes examples using xUnit for its own internal tests, and programmatically specifying measurements, benchmark settings, assertions etc. If the solution I linked to above is not sufficient for people's purposes, I suggest following the practice shown in the NBench source tests.

I hope NBench people don't mind me mentioning it, but Microsoft has an xUnit performance testing project here. Not sure how far it's got, or whether it is still active.

I am going to continue to look at this over the weekend, and if by Monday I have not made further progress, I will release what I do have on NuGet,

[mcquiggd]

@angellaa @RehanSaeed

Another quick update on a Sunday (and I hope I am not spamming this thread!)

I now have a potential solution that works with a custom xUnit Fact attribute, that calls a custom xUnit test runner and test invoker, which makes use of the xUnit framework with an override for the actual test execution, which is a much neater approach.

The main issue I face is, as mentioned above, that NBench currently uses the default, parameterless constructor to instantiate the class containing tests, whereas to provide output via xUnit, a single-parameter constructor has to be called, even if null is passed.

@Aaronontheweb - is there the possibility of altering / overriding the Activator.CreateInstance call that creates the Test Class so that a constructor with a parameter can optionally be called? If we can do that, then we can create a nice integration with xUnit, probably including supporting Theories for multiple test cases, and both Resharper and Visual Studio Test Explorer. It's actually quite neat to be using both xUnit and NBench in the same test suites.

[mcquiggd]

@angellaa @RehanSaeed

Guys, I have upload a new version of my integration between xUnit / NBench / Resharper, using a neater approach with custom xUnit attribute, discoverer, test invoker etc.

https://github.com/Pro-Coded/Pro.NBench.xUnit

@Aaronontheweb - this version does not require any changes to NBench (or xUnit).

(There is just a minor point where test classes are scanned for NBench tests, and all but the targeted method discarded, which is a little inefficient.)

If you can find time to check it out, id prefer some feedback before I release it as a NuGet...

Cheers, David

[Aaronontheweb]

@mcquiggd sorry I've been so quiet on this thread!

This is pretty awesome! I love the idea of piggy backing off of XUnit and NUnit. Definitely a good stop gap short of maintaining a separate plugin.

I'd go ahead and release it as a NuGet package - only thing I think you should add is maybe an end-to-end example. The unit tests do a good job of that though.

[mcquiggd]

@Aaronontheweb

No problem - never enough hours in the day ;-)

I'll create a NuGet tomorrow, too tired now. It would be great if the people interested in this functionality could give it a quick test.

I will also look at adding support for xUnit Theories tomorrow (test parameters supplied by attributes or other data sources) which I think would be rather useful...

David

[mcquiggd]

@angellaa, @RehanSaeed , @Aaronontheweb

Ok guys, NuGet released...

It's a -pre as I have had to use beta releases of xUnit. I'll try to get support for currently released xUnit at some point this week.

https://www.nuget.org/packages/Pro.NBench.xUnit

Install-Package Pro.NBench.xUnit -Pre

Comments / criticisms welcome...

David

[mcquiggd]

@Aaronontheweb

This is specifically to yourself (but others feel free to comment)...

While walking through the NBench code, it became obvious to me that you used what I refer to as a 'known signature delegate' - e.g. Action< BenchmarkContext >, for best performance of invocations without if statements, etc getting in the way.

I wanted to add support for Theories as I think it really expands the scenarios for NBench.

Adding support for xUnit Theories meant adding the possibility of 0-n parameters to be called, potentially of multiple different types; this of course complicated the invocation of the Test method. After trying multiple different approaches, I settled on a DynamicMethod approach to emit a Delegate via IL. This is then wrapped in an Action< BenchmarkContext > so there is very little change to the xUnit NBench invoker from the default one.

But, it is of course vitally important that this does not adversely impact on the accuracy of benchmarks - from my testing, it seems to be perfectly acceptable (especially running in Release configuration, where the IL is optimised). If you notice any detrimental side effects from this approach, shout at me :-)

David

[Aaronontheweb]

@mcquiggd we definitely want to have support for theory-like behavior inside NBench itself as a first class citizens. Being able to run NBench specs with different parameters is a hot topic inside Akka.NET. cc @silv3rcircl3

[mcquiggd]

@Aaronontheweb understood, and makes sense... indeed I started this precisely to get Theories / Scenarios for my core projects.

In the meantime is it worth me continuing with this little project, or do you have plans to release something more NBench native / integrated in the near future?

[mcquiggd]

FYI: NuGet version 1.0.2 released, supports xUnit 2.1 (current stable release) and NBench 0.3.0 (current stable release). Both individual tests [NBenchFact] and paramaterised tests [NBenchTheory] are supported.

NuGet:

https://www.nuget.org/packages/Pro.NBench.xUnit/

Project:

https://github.com/Pro-Coded/Pro.NBench.xUnit

Regards,

David

[mcquiggd]

It's been a while, but thanks to contributor Andrey Leskov, we now have a NetStandard 1.6 version of Pro.NBench.XUnit.

Release to nuget due this weekend...