Improve JRuby results and runnability

[headius]

I'm absolutely sure something is broken in the JRuby results. There's no way we're that slow.

I'd like to help fix up the repository so that it's running appropriately on JRuby. I can't tell what server is being used other than the reference to thin in the gemfile...and thin is definitely not the right server to use with JRuby. I'm also unsure how to get JRuby to run the app. README.md says to just switch to isrubyfastyet/runner and run Rake, but that errors out on JRuby immediately.

Let's discuss how we can fix JRuby on this bench.

[headius]

Ok, it appears that the benchmark simply starts up the script using "rails s", and not installing a server for JRuby. As a result, these benchmarks are running in WEBrick on JRuby (and possibly on other impls. This is not a good representation of performance.

In addition, I do see a commented-out reference to unicorn in the benchmark's gemfile. If MRI is running unicorn versus JRuby running WEBrick, that's an extremely poor representation of performance on JRuby.

I'm attempting to modify the benchmark to use puma on both JRuby and MRI, which will be a better representation. Ideally, the best server for each of MRI and JRuby would be used (maybe unicorn for MRI and torqbox for JRuby).

[headius]

I'm still having trouble getting the benchmarks to run properly, but the following patch should at least get the benchmarks running on puma. I'd still like clarification as to what servers are being used to generate the benchmark results on MRI.

diff --git a/isrubyfastyet/runner/benchmarks/rails/Gemfile b/isrubyfastyet/runner/benchmarks/rails/Gemfile
index 2b9f399..b144142 100644
--- a/isrubyfastyet/runner/benchmarks/rails/Gemfile
+++ b/isrubyfastyet/runner/benchmarks/rails/Gemfile
@@ -38,6 +38,7 @@ gem 'jquery-rails',                       '1.0.19'

 # Use unicorn as the web server
 # gem 'unicorn'
+gem 'puma'

 gem 'capistrano',                         '2.9.0'
 # gem "bcrypt-ruby-maglev-" if defined?(RUBY_ENGINE) && RUBY_ENGINE == "maglev"
diff --git a/isrubyfastyet/runner/benchmarks/rails/benchmark_helper.rb b/isrubyfastyet/runner/benchmarks/rails/benchmark_helper.rb
index e194019..46bbb98 100644
--- a/isrubyfastyet/runner/benchmarks/rails/benchmark_helper.rb
+++ b/isrubyfastyet/runner/benchmarks/rails/benchmark_helper.rb
@@ -25,7 +25,7 @@ module BenchmarkHelper
   end

   def start_production_server(bash)
-    bash.puts "RAILS_ENV=production #{ENV['RUBY_COMMAND']} script/rails s -p3009 >& /dev/null &"
+    bash.puts "RAILS_ENV=production #{ENV['RUBY_COMMAND']} script/rails s puma -p3009 >& /dev/null &"

     timeout_start = Time.now
     sleep 0.2 until server_running? || (Time.now - timeout_start > 120)

[brianhempel]

Eww, I'm sorry you had to try to follow that undocumented mess of the benchmark structure.

I'm going to open separate issue for getting the suite runner to start on JRuby (issue #6). I was doing some crazy forking business, but the system method may actually be sufficient.

[brianhempel]

As far as puma goes, I want to jump into the JRuby irc and discuss that soon. I originally stuck with Webrick because I wanted this to be a Ruby benchmark—more Ruby in the stack rewards implementations that better handle different kinds of Ruby code.

However, I realize that many people are asking the question, "I have x amount of hardware, what Ruby implementation should I use to maximize the amount of rails traffic I can handle with it?" I am leaning towards adding another benchmark to test maxed-out multithreaded production performance. Thermal issues will be the trouble there.

[zbrock]

I agree that Puma would be a lot more realistic. Would love to see this added. It would also be great to see this run on Linux on a standard AWS instance. That would be a useful real world benchmark!