noahgibbs / rsb

Rails Simpler Bench - a simple Rails app, with a variety of requests and Ruby versions that it can be tested with
42 stars 6 forks source link

Add a CLI runner and ensure benchmarks keep working by testing in TravisCI #6

Closed eregon closed 5 years ago

eregon commented 5 years ago

Also includes the commit from https://github.com/noahgibbs/rsb/pull/5

The new runner is more convenient (tweaking values is trivial, and it's self-documenting) when testing just one Ruby and analyzing performance, and when running the benchmark in an automated manner for just one Ruby implementation at a time.

$ runners/current_ruby_cli.rb -h
Usage: runners/current_ruby_cli.rb [options] APP SERVER
  APP: rack, rails
  SERVER: webrick, puma, thin, unicorn, passenger
        --wrk-binary STRING                      Default: "wrk"
        --wrk-concurrency INTEGER                Default: 1
        --wrk-connections INTEGER                Default: 100
        --wrk-script-location STRING             Default: "./final_report.lua"
        --[no-]wrk-close-connection              Default: true
        --warmup-seconds INTEGER                 Default: 5
        --benchmark-seconds INTEGER              Default: 180
        --before-worker-cmd STRING               Default: "bundle install"
        --ruby-subprocess-cmd STRING             Default: "BEFORE_WORKER && ruby SUBPROCESS_SCRIPT JSON_FILENAME"
        --wrap-subprocess-cmd STRING             Default: "COMMAND"
        --json-filename STRING                   Default: "/tmp/benchlib_7224.json"
        --wrk-subprocess STRING                  Default: "/home/travis/build/eregon/rsb/wrk_subprocess.rb"
        --rack-env STRING                        Default: "production"
        --bundle-gemfile STRING                  Default: "Gemfile.2.6.3"
        --bundler-version STRING                 Default: nil
        --port INTEGER                           Default: 4321
        --timestamp STRING                       Default: nil
        --url STRING                             Default: "http://127.0.0.1:PORT/static"
        --out-file STRING                        Default: "data/rsb_output_TIME.json"
        --verbose INTEGER                        Default: 1
        --server-cmd STRING                      Default: nil
        --server-ruby-opts STRING                Default: nil
        --server-pre-cmd STRING                  Default: nil
        --server-kill-command STRING             Default: nil
        --server-kill-matcher STRING             Default: nil
        --[no-]suppress-server-output            Default: false
        --[no-]check-url                         Default: true

The TravisCI config ensures both the rack and rails apps work on 2.6.3 and 2.0.0. Here is the latest run: https://travis-ci.org/eregon/rsb/builds/587827779

@noahgibbs What do you think? Could you enable TravisCI on this repo?

noahgibbs commented 5 years ago

Travis seems reasonable. I haven't used it here yet, but it's high time.

I notice that you changed the sense of "no_check_url" to "check_url", which is good practice and looks good.

I'm fine with adding new runners.

The new Gemfile-2.0.0 is reasonable. I have separate ones for p0 and p648, but that's a relic and can be removed. I'm in the process of adding dynamic Gemfile generation (the commits are in, but apparently it's broken for you? Getting there...) But no harm in adding another Gemfile in the mean time.

eregon commented 5 years ago

Thanks for merging!

I added Gemfile-2.0.0 simply because that's what gets used when using RUBY_VERSION to select the Gemfile. It would probably be simpler to use RUBY_VERSION to select the Gemfile for other runners as well, or maybe it's not necessary anymore with Gemfile generation.

noahgibbs commented 5 years ago

There is a lot less need for it with Gemfile generation - if that basically works for you then the only reason to specify a Gemfile is if you want something specific (e.g. pry, specific gem for speed-testing, specific gem version for compatibility.)