search

dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.57k stars 970 forks source link

Graph: Combined Jitter Scatter-plot and Box-plot #275

Open redknightlois opened 8 years ago

redknightlois commented 8 years ago

With all the data that any benchmark gets as results and the amount of times operations tend to be executed, its not a bad idea to also allow outputting this kind of graph

Combined Jitter Scatter plot and Box plot

The explanation of the graph is here: https://www.pvk.ca/Blog/2012/07/03/binary-search-star-eliminates-star-branch-mispredictions/

dbgoytia commented 2 years ago

Hey @AndreyAkinshin ! I was wondering if this issue is still relevant? I'd like to contribute, although I might need a bit of help getting started

AndreyAkinshin commented 2 years ago

@dbgoytia yes, it's still relevant. At the moment, BenchmarkDotNet doesn't have its own plotting subsystem. The only way to generate plots is [RPlotExporter]. It uses locally installed R and a custom R script that could be found here. Thus, the only way to support a new kind of plots is to add the corresponding changes in the R script (no C#, sorry).

P.S. It would be nice to have own plotting system so that we can draw charts without installed R. However, we don't want to add any additional dependencies to other NuGet packages. Thus, we should implement such a plotting logic from scratch. It's a huge tasks that require a lot of efforts, so we don't have it in the roadmap for the nearest future. Also, it makes sense to implement it on the perfolizer side rather than in BenchmarkDotNet.

FlatlinerDOA commented 7 months ago

Can I suggest a separate Nuget package is created to support it BenchmarkDotNet.Exporters.Plot or something, and use a library like ScottPlot at least for basic box plots - https://scottplot.net/cookbook/5.0/

AndreyAkinshin commented 7 months ago

@FlatlinerDOA sounds good to me. While I'm still against adding NuGet dependencies to the main package without a good reason, we have no limitations on creating new packages with any dependencies.

FlatlinerDOA commented 7 months ago

@AndreyAkinshin I've got a basic Bar Plot outputting for now in a separate package called BenchmarkDotNet.Exporters.Plotting. Is your preference to get all plots supported first (in one large PR). Or just get a preview package set up and add plot types one by one?

AndreyAkinshin commented 4 months ago

@FlatlinerDOA thanks again for the PR and sorry for such a long delay with merging. It's better to add plots one-by-one. I will include the new NuGet package in the upcoming release.

FlatlinerDOA commented 3 months ago

@AndreyAkinshin no worries, I have created two PR's: One #2611 is a small to fix a bug in the result filtering, and #2614 for adding Box Plots.