Discussion: Reporting requirements

ppanyukov commented 9 years ago

(Don't know if this is the best place/way to talk about it but lets do this for a start. Perhaps wiki or something else might be better?)

Let's have a place to collect and discuss our requirements for reporting. This will drive whatever dev work we need to do to get these done. Feel free to edit / comment etc.

One thing is certain: there are different use case which may require different kind of reporting.

For now, briefly:

1. Ad-hoc runs for visual inspection of results

(current use case)

Run the benchmarks in ad-hoc manner;
See the results on screen;
Possibly copy/paste into some blog in Markdown format;
The main goal here is to compare results between each other on-screen.

Key here is:

Have results displayed so they are easily viewed on screen;
Numbers should be easily comparable;
Want to see what's faster/slower and by how much;
Compact Markdown-friendly form.

(this is catered by the current reporting, with some enhancements we could do there)

2. Spreadsheet analytics

Run lots of different benchmarks, possibly on multiple systems;
Collate all results;
Load into Excel whatever and analyse and chart there.

Key here is:

Output format friendly for Excel / other analytics tools, means probably CVS;
Comparison between results of the same benchmark on the same system;
Comparison between results from runs on different systems.

3. Fully automated runs, dashboards etc

This extends on "Excel" case and adds automation and scale.

Run many benchmarks on many systems;
Do this on regular basis;
The benchmarks themselves may be some code which changes over time;
The execution environments may change over time (patches, infrastructure changes etc);
The benchmarks are scheduled to run on regular basis in automated mode;
Results are collected and published to reporting/analytics systems (e.g. Splunk, PowerBI);
Dashboards are created to see trends over time, alerts for degradations etc.

Keys are:

Automation-friendly;
Collect and keep the results;
Timestamps in results are a must;
Friendly for ingestion to target analytics systems;
Friendly for dashboard-building;
Friendly for alerting;

These are probably the main ones for a start I can think of now.

AndreyAkinshin commented 9 years ago

@ppanyukov, thank you for your ideas. Of course, all kind of reporting are important. However, just for now I suggest to focus on ad-hoc runs for visual inspection of results, it is the most actual problem. After that, we can think in detail about export formats.

ppanyukov commented 9 years ago

Yes I agree. We (or rather I) will improve the adhoc first :)

These are from the use cases I have in real life and are here to keep the big picture in mind and understand why certain reports need to have different formatting requirements. For example, why number alignment and using the same units in ad-hoc is important.

mattwarren commented 9 years ago

I agree with all the comments above, basically get 1 (Ad-hoc runs for visual inspection of results) sorted out first and then do 2 (Spreadsheet analytics) and 3 (Fully automated runs, dashboards etc)

The majoriy of work needed for 2 and 3 can be handled by allowing the results from a command line run to be exported to .csv, .json and/or .xml

mattwarren commented 9 years ago

Want to see what's faster/slower and by how much;

For this, maybe there could be a mode where the results are sorted, faster -> slowest, rather than in alphabetical order?

AndreyAkinshin commented 9 years ago

@mattwarren,

For this, maybe there could be a mode where the results are sorted, faster -> slowest, rather than in alphabetical order?

It is a good point, but I think that we should use the alphabetical order as a default because it allows to easily compare results in different environments. However, it would be great to create a command line argument for different display orders.

mattwarren commented 9 years ago

but I think that we should use the alphabetical order as a default

yeah that makes sense

AndreyAkinshin commented 9 years ago

@ppanyukov, @mattwarren, I have committed some stuff for the future reporting system, see https://github.com/PerfDotNet/BenchmarkDotNet/commit/e5f7484f4d0f7acd9a23d91cbcc6615c8fa8154b What do you think?

mattwarren commented 9 years ago

I just had a quick look at the code, looks good.

It's a nice way of seperating out the different behaviours csv, markdown etc. Should make it nice and easy to extend it in the future, for instance json

Will it be a configurable via a command-line argument?

AndreyAkinshin commented 9 years ago

Will it be a configurable via a command-line argument?

Yep, I will add the configuration support soon.

ppanyukov commented 9 years ago

I had a look at the change on my phone :)

Broadly looks fine to me, with a couple of comments:

obviously, as was already mentioned, some command line options to specify output type(s) would be nice;
I understand that csv and md outputs are currently created regardless. Obvious concern is not everybody would want it. More subtly the whole thing may fail after running the for a while if there are no permission to write the file. Sort of brings us back to command line args. A workaround is to try create output file before we run any benchmarks and fail fast right then rather than at the very end. Probably something to think about even with command line options.

Regarding the command line args, ideally would be nice to specify multiple output types I guess. For example you might have:

Run.exe --mdOut foo.md --csvOut bar.csv

For parsing command line options I found this package which worked well for me: CommandLineParser.

You can see sample usage here: https://github.com/ppanyukov/WebReplay/blob/master/src/WebReplay/CommandLineOptions.cs

You can see the kind of options / help it produces here: https://github.com/ppanyukov/WebReplay/blob/master/readme.md

Sent from my iPhone

On 2 Sep 2015, at 13:41, Andrey Akinshin notifications@github.com wrote:

Will it be a configurable via a command-line argument?

Yep, I will add the configuration support soon.

— Reply to this email directly or view it on GitHub.

AndreyAkinshin commented 9 years ago

obviously, as was already mentioned, some command line options to specify output type(s) would be nice;

Ok, it will be done.

if there are no permission to write the file

Then we have a problem: BenchmarkDotNet generates whole projects and compiles it. Regardless defaults, command line args is a good idea.

ppanyukov commented 9 years ago

Ah ok, ignore the permissions bit then. Although with users being able to specify arbitrary output paths the problem may return. A minor issue in the grand scheme of things.

Sent from my iPhone

On 3 Sep 2015, at 00:31, Andrey Akinshin notifications@github.com wrote:

obviously, as was already mentioned, some command line options to specify output type(s) would be nice; Ok, it will be done.

if there are no permission to write the file

Then we have a problem: BenchmarkDotNet generates whole projects and compiles it. Regardless defaults, command line args is a good idea.

— Reply to this email directly or view it on GitHub.

AndreyAkinshin commented 8 years ago

@ppanyukov, I think, we have a nice reporting system now. It's flexible, we have a lot of standard exporters, you can also write you own classes which export benchmark results to any format. So, I close this issue. If you have some additional specific requests, please, create a new issue.

dotnet / BenchmarkDotNet

Discussion: Reporting requirements #19