Open BrianHicks opened 6 years ago
Comment by zwilias Thursday Mar 09, 2017 at 21:41 GMT
I think it might make sense to consider the use case of comparing a single baseline with multiple variations of implementation.
compare : String -> Benchmark -> List Benchmark -> Benchmark
for example, might make sense.
In my specific use case, I'm generally not very interested in only having a single datapoint per function I'm testing, but rather, I want to compare performance over a series of datapoints, each with possible variations. (This ties in to #3)
Something along the lines of:
compareSeries : String -> (a -> Benchmark) -> List (a -> Benchmark) -> List a -> Benchmark
A use case here would be testing insertion in various dict implementations, with core's dict as the baseline, creating trees of size x
.
What might make this tricky is the expected behaviour of such a construction, especially if the Benchmark
s are Group
s or even Compare
s themselves. Basically, the logic you've described would apply, with the addition of allowing comparisons of a single baselines to multiple implementations.
Use case here is an extension of the one mentioned above - each benchmark could be a group where insertions on trees with different key types are tested.
Essentially, I'd rather express that I'm comparing a series of inputs, than express I'm creating a series of comparisons.
Issue by BrianHicks Monday Feb 13, 2017 at 16:11 GMT Originally opened as https://github.com/BrianHicks/elm-benchmark/issues/6
Right now
compare
only compares two single benchmarks (that is,Benchmark
s created withbenchmark
throughbenchmark8
.) We need to be able to compare any validBenchmark
. I suggest the following:Group
s (that is,Benchmark
s created withdescribe
) we will zip the elements usingcompare
, and modify the inner names. If one list is longer than the other, we will only zip the common elements and drop the rest. This will return aGroup
named "{group a} vs {group b}" whose elements areCompare
s named "{element a name} / {element b name}" with children named "{group a}" and "{group b}". This logic recurses if any child elements are groups themselves.Compare
s (that is,Benchmark
s created withcompare
) we will combine as if they're groups with exactly two elements each.Group
s with two elements.