winr

A quick and lightweight benchmarking tool for Ruby

Goals

Provide a simple way to benchmark code
Easy to configure and compare results

Overview

There are several tools for benchmarking Ruby code. Unfortunately, it's hard to remember the way to set up the benchmarks, what format to use to represent what to test, and how to perform the testing. One of the most popular tools used for benchmarking Ruby code is benchmark-driver, which was used heavily by the Ruby 3x3 team. It uses three levels of abstraction and is powerful, but there is a lot of inconsistency in naming conventions and using the tool.

The idea of winr is to offer an easy to configure benchmarking tool that is easy to use. There are three levels of abstraction:

environments - An environment is an optional top-level container for running code. It can represent different versions Ruby, or different setup options that will be shared by any nested contexts or tasks.
contexts - A context represents an optional mid-level container for running code. Any context configuration is shared by any contained tasks.
tasks - A task represents a unit of work to be tested, within its optional containing context and optional containing environment.

Within an environment, context, or task, the following can be defined:

name - The name of the element, to use when displaying output
begin - Code that should be run before any contained elements
end - Code that should run after any contained elements
script - Code that is contained within a task
loops - The number of times that a task's script should be executed
warmup - An optional top-level setting that is used when the number of loops has not been defined. This value is the number of seconds to run the task before determining the number of loops that have run. The default value is 3 seconds, meaning that the number of loops is not defined, the task will be allowed to run for 3 seconds and the number of iterations during this time will be used.

Examples

Configuration information is stored in a Ruby file, like the following:

$ cat test/sum.rb

{
  environments: [
    {
      name: "Shiny new Ruby",
      command: "/opt/ruby-3.2.0/bin/ruby",
    },
    {
      name: "Old trusted Ruby",
      command: "/opt/ruby-2.7.6/bin/ruby",
    },
  ],
  contexts: [
    {
      begin: <<~"|",
        require "digest/md5"

        max = 1e5.to_i
      |
    },
  ],
  tasks: [
    {
      name: "array splat",
      script: <<~"|",
        ary = [*1..max]
        sum = ary.sum
      |
    },
    {
      name: "times",
      script: <<~"|",
        sum = 0
        max.to_i.times {|n| sum += n }
      |
    },
  ],
  warmup: 3,
}

The benchmark is run as follows:

$ winr test/sum.rb

┌──────────────────┬───────────────────────────────────────┐
│ Shiny new Ruby   │                Results                │
├──────────────────┼───────────┬─────────────┬─────────────┤
│ array splat      │   2.99  s │ 768.19  i/s │   1.30 ms/i │
│ times            │   7.86  s │ 292.33  i/s │   3.42 ms/i │
└──────────────────┴───────────┴─────────────┴─────────────┘

┌──────────────────┬───────────────────────────────────────┐
│ Old trusted Ruby │                Results                │
├──────────────────┼───────────┬─────────────┬─────────────┤
│ array splat      │   2.96  s │ 777.46  i/s │   1.29 ms/i │
│ times            │   7.73  s │ 297.19  i/s │   3.36 ms/i │
└──────────────────┴───────────┴─────────────┴─────────────┘

┌──────────────────┬─────────────────────────────────────┐
│ Rank             │             Performance             │
├──────────────────┼─────────────┬──────────────┬────────┤
│ array splat      │ 777.46  i/s │   fastest    │ 2/1/1  │
│ array splat      │ 768.19  i/s │ 1.01x slower │ 1/1/1  │
│ times            │ 297.19  i/s │ 2.62x slower │ 2/1/2  │
│ times            │ 292.33  i/s │ 2.66x slower │ 1/1/2  │
└──────────────────┴─────────────┴──────────────┴────────┘

The following screenshot shows the output (notice the units and color), when the value of max is changed to 1e3.

Install

Install via rubygems with:

gem install winr

Options

$ winr -h

usage: winr [options] <dir ...>
    -c, --[no-]color                 Enable color output (default is true)
    -d, --debug                      Enable debug mode
    -h, --help                       Show help and command usage
    -i, --iterations <count>         Force the number of iterations for each task
    -r, --reverse                    Show contexts vertically and tasks horizontally
    -s, --stats                      Comma-separated list of stats (loops, time, ips, spi)
    -v, --verbose                    Show command, version details, and markdown backticks

Available statistics:

  ips      iterations per second
  loops    number of iterations
  spi      seconds per iteration
  time     time to run all iterations

License

This software is licensed under terms of the MIT License.

shreeve / winr

readme