[NeedHelp] Running under nunit?

[ig-sinicyn]

Hi!

I'm trying to use BenchmarkDotNet inside NUnit to get something alike NBench does. I've started with existing example, but it fails to run with UnauthorizedAccessException, path "C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 14.0\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW\Assertions.log"

Full stack:

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
   at System.IO.StreamWriter..ctor(String path, Boolean append)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run(IList`1 benchmarks, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run[T](IConfig config)
   at CodeDrafts.PerfTests.Assertions.Test() at PerfTests\Assertions.cs:line 37

Some guide or working samples will be greatly appreciated:)

[mattwarren]

hmm, I wonder what NUnit is doing differently from XUnit (which all our integration tests are written in).

Can you post the C# code needed for a full repo and we can take a look?

[ig-sinicyn]

@mattwarren: Sure, https://gist.github.com/ig-sinicyn/f75c982913409e8d9567

Also, I need some recommendations for the following use case:

I am going to write perf test as a competition. There will be a dumb but proofed-to-work solution (baseline) and all other implementations will be compared to the baseline. All perf assertions will be relative, e.g. 95 percentile will take no more than 1.2x time, 2.0x allocations etc.

Also I'm going to hide all magic into a base class and derive all perf tests from it.

Looking for any recommendations if possible:)

[mattwarren]

Sounds interesting, I know that this is one thing that BenchmarkDotNet lacks, so it'll be nice to see what you come up with.

It would also be something that we could include in BenchmarkDotNet as an extension, i.e. a separate "BenchmarkDotNet.Nunit" NuGet package, so consider sending a PR once you have something working. We'd like to have better OOTB Unit Test runner support, as all we have at the moment is the example you linked to

I am going to write perf test as a competition. There will be a dumb but proofed-to-work solution (baseline) and all other implementations will be compared to the baseline. All perf assertions will be relative, e.g. 95 percentile will take no more than 1.2x time, 2.0x allocations etc.

There's a few features in BenchmarkDotNet that might help you:

1. Annotated your baseline with [Benchmark(Baseline = true)] and you get an extra column with the results scaled, so that the baseline is at 1.0
2. For memory allocations see my blog post. Note that currently this only works if you compile BenchmarkDotNet from source, but soon we will have a BenchmarkDotNet.Diagnostics NuGet package to make it easier.

[ig-sinicyn]

@mattwarren ok, I'll do PR if I'll manage the entire idea to work:)

It turns out there's something wrong with working directory. VS test runner for NUnit sets C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 14.0\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW

At least three solutions possible:

1. Figure out howto change working dir in NUnit (some setup in fixture?) 2, Explicitly set the output path for BenchmarkRunner (i it possible)?
2. Hackery: just add Environment.CurrentDirectory = Path.GetTempPath(); as a first line of the test.

The third one worked for me and I'm going to leave it as is for now. Thanks!

UPD Oh, that's NUnit' vs adapter bug. It was born to fail, as for me:)

UPD2 And Environment.CurrentDirectory = TestContext.CurrentContext.TestDirectory; will be better, I guess.

[mattwarren]

ok, I'll do PR if I'll manage the entire idea to work:)

That would be great!

At least three solutions possible:

1. Figure out howto change working dir in NUnit (some setup in fixture?) 2, Explicitly set the output path for BenchmarkRunner (i it possible)?
2. Hackery: just add Environment.CurrentDirectory = Path.GetTempPath(); as a first line of the test.

The third one worked for me and I'm going to leave it as is for now. Thanks!

Yeah we've come across this issue before, see https://github.com/PerfDotNet/BenchmarkDotNet/issues/66#issuecomment-170874161.

I need to take another look at it but from what I remember the problem is that we use Directory.GetCurrentDirectory (which is the same as Environment.CurrentDirectory) but as you've seen it isn't always robust.

[mattwarren]

@ig-sinicyn you might actually be able to take the NUnint integration one step further, see the links below for some ideas:

• http://www.dotnetalgorithms.com/2016/03/nbench-performance-testing-nunit-resharper-integration/
• https://github.com/angellaa/dotnetalgorithms/blob/master/NBench/TestingWithNUnit/PerformanceTestStuite.cs

[ig-sinicyn]

@mattwarren

Well, I finally get quick-and-dirty prototype to work: https://gist.github.com/ig-sinicyn/995cbaf64cf42d570b6d

Output:

Test Name:  TestNotSlowerThan10
Result Message: 
Bench PerfAssertions_SlowBenchmark_Dry is slower than 2x baseline. Actual: 8,58x

It fails just as planned:)

I'll going to improve it (in particular there should be a config paramerer and percentile should be configurable too), but this for later.

[mattwarren]

Nice work!

BTW does the NUnit runner capture the BenchmarkDotNet output?

[ig-sinicyn]

@mattwarren nope, #95.

Can be fixed easily by custom listener calling Debug.WriteXyz()/Trace.WriteXyz().

[ig-sinicyn]

@mattwarren Oops, I was wrong multiple ways:)

Bad news: custom ILogger redirecting to Trace does nothing. Good news: Console output is collected, it's just hidden under Output link in the test explorer.

Current console output is slightly noisy, so I'll try to disable default console logger and add custom MarkdownExporter wrapping ConsoleLogger instead.

[mattwarren]

Current console output is slightly noisy, so I'll try to disable default console logger and add custom MarkdownExporter wrapping ConsoleLogger instead.

That's a good idea, put the summary table in the test result and then the full output in the "Output" link in test explorer.

[ig-sinicyn]

@mattwarren As far as I know there's no good way to display large portions of text inside the VS Test Explorer window, Also text from Summary table will not fit into message field in the Test Explorer panel.

So I'll do the following:

1. Remove the default console logger.
2. Capture all output into AccumulationLogger
3. Dump summary into console using MarkdownExporter.Default.ExportToLog(summary, ConsoleLogger.Default);
4. Write some blank lines
5. Dump all text from the AccumulationLogger back into console.

This way there will be Summary data at top and all other details at bottom. The sad part: VS currently does not use monospace font in test output window and there's no easy way to fix it. Proofs: https://visualstudio.uservoice.com/forums/121579-visual-studio-2015/suggestions/6126561-fix-test-explorer-output-monospace-font-and-outpu http://stackoverflow.com/questions/17671054

[mattwarren]

Also text from Summary table will not fit into message field in the Test Explorer panel.

Oh yeah I forgot about that, our summary table can get quite wide!

So I'll do the following: ....

Sounds like a good plan

The sad part: VS currently does not use monospace font in test output window and there's no easy way to fix it. Proofs:

That has always annoyed me!! I just end up copy-pasting the output into Notepad++

[mattwarren]

@ig-sinicyn are you happy for this issue to be closed?

I think that any actions are covered elsewhere, such as #155

[ig-sinicyn]

@mattwarren

Yes, of course. Forgot about this one.