davezych / shience

A .NET port(ish) of Github's Scientist library. (https://github.com/github/scientist)
MIT License
9 stars 1 forks source link

Add benchmarking methods #33

Open davezych opened 8 years ago

davezych commented 8 years ago

It might be useful to provide benchmarking methods to Shience. This would allow the user to wrap Shience around a single method, and publish the runtime stats just to track the average runtime over time for a method.

A simple solution would be to allow the candidate to be null when calling Test, which would mean only the control would run and there would be publish data for just the control available.

A "better" solution would be something like:

experiment.Benchmark(() => SomeCall())
                  .PublishTo((e) => Results(e))
                  .Execute();
MovGP0 commented 8 years ago

i guess that the benchmarking will require some internal singleton thread-save hashtable to keep track of the statistics.

also, we would have to define what kind of statistics we need. min, max, average, median, standard deviation, percentiles, and the like comes to my mind.

maybe the user should keep track of the statistics in the publish method himself, since not every user will need all statistics.

MovGP0 commented 8 years ago

the LinqStatistics package could be useful.