Provide support for running using an existing .NET exe

[agocke]

So, here's my scenario:

I would like to use Benchmark.NET to measure the performance of the Roslyn compiler end-to-end, meaning my benchmark is basically csc -noconfig @repro.rsp.

I'll grant you, this isn't an obvious fit for BenchmarkDotNet, but technically I'm still benchmarking .NET code and there are a lot of things, like providing easy swapping of runtimes, that BenchmarkDotNet is great for. Perhaps most importantly, I don't really want to redo all the statistical analysis that's already built into BenchmarkDotNet.

So anyway, the problem is that I essentially need to do a process launch. The normal operation of BenchmarkDotNet would do two process launches, but I've managed to kind of get this working using the InProcess running mode (and manually working around the default timeout -- why is there a timeout on the InProcess mode??), but this is really hacked up.

Here's my proposal:

We should add support for the BenchmarkAttribute on properties and fields of type ProcessStartInfo. The contract here is that ProcessStartInfo would contain an arguments array that includes as the first argument the path to the managed EXE. BenchmarkDotNet would then plug in a process executor that would slot in the appropriate runtime and runtime options and then execute the given StartInfo. Other things would work somewhat analogously.

This would allow me to keep all my code in the external EXE, which is much easier to compile standalone, and also easily allow me to swap out runtimes.

[shargon]

Should be fine if we can do all on the same execution (without new process). For visual studio profile performance is very useful