Diagnostic improvements

[AndreyAkinshin]

@mattwarren, I want the following list of features:

• [X] 1. I want be able to add a diagnostic class with the AddDiagnoster method in manual mode (automatic loading should work in audo mode).
• [X] 2. I want a special diagnostic BenchmarkIterationMode for diagnostic. It should be run before the pre-warmup stage.
• [ ] 3. Now I have troubles with diagnostic of x86 benchmark from the x64 host application. Can we fix it?
• [x] 4. It would be cool to split the current diagnostic class to source diagnoster (IL and ASM) and runtime diagnoster (GC, Segments, and so on).
• [x] 5. We should cover the diagnostics by unit tests.

What do you think?

[AndreyAkinshin]

I have tried to implement tasks 1 and 4. Please, review (https://github.com/PerfDotNet/BenchmarkDotNet/commit/28083486f9423229545969617cab4b05909ba346). Now you can add specific diagnostic with a command like arguments like -d=source,runtime (d means diagnostic, target diagosers will be matched by name).

[mattwarren]

I have tried to implement tasks 1 and 4. Please, review (2808348).

Thanks for doing that, at first glance it looks good, but I'll have a more of a look later on today.

2 - I want a special diagnostic BenchmarkIterationMode for diagnostic. It should be run before the pre-warmup stage.

Just curious, what's the reason for this? Is it so the diagnostic doesn't interfere with calculating the invokeCount

3 - Now I have troubles with diagnostic of x86 benchmark from the x64 host application. Can we fix it?

Yeah that's a problem with the CLRMD library, it doesn't want to analyse a 32-bit process from a 64-bit host app. I'll take a look, I think I can fix it by having a small shim 32-bit process that can then launch the 32-bit benchmark.

[AndreyAkinshin]

Thanks for doing that, at first glance it looks good, but I'll have a more of a look later on today.

Ok. I just implemented basic concepts. Please, refactor the diagnostic classes at its discretion.

Just curious, what's the reason for this?

Theoretically, it can affects the warmup process. We should try to avoid any additional logic between our runs. So, I suggest to create a special stage: diagnostic. We can run it directly without hacks like if (line.StartWith("// Warmup") .... The source code will be more clear.

I'll take a look, I think I can fix it by having a small shim 32-bit process that can then launch the 32-bit benchmark

Sounds cool.

[mattwarren]

Theoretically, it can affects the warmup process. We should try to avoid any additional logic between our runs. So, I suggest to create a special stage: diagnostic.

This is fine, I'll can add an extra step and only run it if we need diagnostics, i.e. BenchmarkState.Instance.IterationMode = BenchmarkIterationMode.Diagnostics;

We can run it directly without hacks like if (line.StartWith("// Warmup") .... The source code will be more clear.

I had a look at this, the main problem is that we are in different processes. The diagnostics are run from the main process, where as BenchmarkIterationMode is set in the process we spawn. So the only real way they have to communicate between the 2 is by looking at the console output, or am I missing something?

[mattwarren]

Now being tracked via #160, so closing this old issue