`compare!(order: :baseline)` option

[casperisfine]

This is just a quick proof of concept to gather some feedback, I totally realize it's likely not desirable to change the existing behavior, but I'm willing to do the work to put this behind an option of some sort.

Use case.

Most of the time when I use benchmark-ips, it looks like this:

Benchmark.ips do |x|
  x.report("original") {}

  x.report("opt_1") {}
  x.report("opt_2") {}
  # ...
  x.compare!
end

Giving a result like:

Warming up --------------------------------------
             original    96.197k i/100ms
             opt_1       63.076k i/100ms
             opt_2      250.949k i/100ms
Calculating -------------------------------------
             original 953.227k (± 0.9%) i/s -      4.810M in   5.046288s
             opt_1    653.881k (± 5.0%) i/s -      3.280M in   5.028459s
             opt_2      2.494M (± 0.3%) i/s -     12.547M in   5.031949s

Comparison:
             opt_2:     2493580.5 i/s
             original:   953227.3 i/s - 2.62x  (± 0.00) slower
             opt_1:      653881.1 i/s - 3.81x  (± 0.00) slower

But maybe it's just me, I find this output a bit confusing, because what I want to know is how much slower/faster each optimization is compare to "the baseline", which generally is the first registered x.report.

So something like this:

Warming up --------------------------------------
             original    93.207k i/100ms
             opt_1       61.370k i/100ms
             opt_2      255.237k i/100ms
Calculating -------------------------------------
             original    933.290k (± 0.8%) i/s -      4.754M in   5.093679s
             opt_1       642.273k (± 4.0%) i/s -      3.253M in   5.072394s
             opt_2         2.538M (± 0.2%) i/s -     12.762M in   5.029163s

Comparison:
             original:   933290.0 i/s
             opt_2:     2537580.1 i/s - 2.72x  (± 0.00) faster
             opt_1:      642272.7 i/s - 1.45x  (± 0.00) slower

Which is pretty much what I implemented in a quick and dirty way.

Possible API

So if this feature were to be accepted, what would be the syntax for it?

I'm thinking that it could be mark by using x.baseline instead of x.report ?:

Benchmark.ips do |x|
  x.baseline("original") {}

  x.report("opt_1") {}
  x.report("opt_2") {}
  # ...
  x.compare!
end

[kbrock]

neat idea.

Just having a baseline option (where we assume the first is the baseline?) may be minimal changes.

you would just need:

      best = entries.shift if baseline?
      sorted = entries.sort_by{ |e| e.stats.central_tendency }.reverse
      best ||= sorted.shift

and the changes to handle faster/slower

[nateberkopec]

Yeah I feel like most of the time you just put the baseline first right?

Seems like an option like x.compare!(order: :baseline) would work?

[casperisfine]

I feel like most of the time you just put the baseline first right?

That's my case at least yeah.

an option like x.compare!(order: :baseline) would work?

Sounds good to me, I'll try that.

[casperisfine]

Done.

I couldn't really see how to test it however, as the current test don't really cover the rendering.

[casperisfine]

@evanphx done.

[kbrock]

Not sure if is more clear to introduce another report: baseline(:original) {} (this current PR does this) Or just introduce a variable. report('original') {} ; self.baseline='original' (and that variable, like warmup could come in any of a number of ways)

[kbrock]

Yeah I feel like most of the time you just put the baseline first right?

Seems like an option like x.compare!(order: :baseline) would work?

ooh. I think I spoke too soon. So you do not introduce baseline block or variable at all, just order. That is potentially nice.

code looking nice @casperisfine

[casperisfine]

Thanks!

[nateberkopec]

Release coming today or tomorrow