search

petabridge / NBench

Performance benchmarking and testing framework for .NET applications :chart_with_upwards_trend:
https://nbench.io/
Apache License 2.0
532 stars 46 forks source link

Discovery + Benchmark Execution + Metric Collection + Reporting Engine Redesign #226

Open Aaronontheweb opened 6 years ago

Aaronontheweb commented 6 years ago

With the advent of .NET Core, the process of running benchmarks is much more complicated than it used to be given that we have to cover a much larger surface area of runtimes, run time versions, and platforms.

The design that we implemented to address this when .NET Core 1.0 was on the horizon is frankly pretty weak and, as it turns out, never worked properly in the first place. Therefore, we should use this as an opportunity to reassess our design and build something that can properly support a much larger platform / runtime surface area.

Moreover, I want to support something that is much more extensible than what we have today in terms of both benchmark execution, discovery, and metrics collection.

Going to keep this issue open as a means of organizing some of the designs for accomplishing this.

Aaronontheweb commented 6 years ago

nbench execution flow 6-8-2018

Should be able to click for a full size image, but this is how the execution flow essentially works today, minus some features that we haven't implemented such as #220. However, the way this is expressed via the NBench SDK isn't all that clear and we intend on breaking up the execution model to allow for customization with each of these parts much more easily in the future.