Trouble in mixed MRI/JRuby environment

wincent commented 14 years ago

I noticed my Steak/Cabybara/Culerity/Celerity specs broke the other day with this commit:

http://github.com/rspec/rspec-core/commit/c467c03ecac4ed7310c6fa27e551d7f19e93ae1a

Specifically they were bombing out (hanging or otherwise failing) as soon as I hit any spec involving JRuby (Celerity).

After much digging around I found the following:

if running using "rspec foo", everything worked fine
if running using "bundle exec rspec foo", the errors appeared
likewise errors appeared, as of rspec-core commit c467c03, when running using "rake spec:acceptance"

Turns out that the reason the last two fail is because both of them set RUBYOPT to include the path of the Bundler "lib" directory (harmless), and also "-rbundler/setup" (not so harmless, in the context of JRuby). RSpec doesn't directly set RUBYOPT, but by calling "bundle exec" it effectively causes Bundler to do it. The RSpec behavior isn't wrong, of course; matching the behavior of "bundle exec" is evidently the right thing to do.

The problem, however, arises when JRuby comes into the picture. Everything is running inside MRI until it comes time to do some JavaScript testing, and at that point Culerity (Ruby) executes Celerity (which is JRuby and has to be because only by running in the JVM can it use HtmlUnit, which is Java). At this point the presence of RUBYOPT starts to cause problems because the JRuby process trips up when it tries to evaluate "bundler/setup", for reasons which I don't yet fully understand.

Obviously, "bundler/setup" ends up calling Bundler.setup, and it seems that unless the JRuby environment contains all the gems that were present in the Bundler-managed MRI environment, it will evaluate the Gemfile and end up throwing an exception and a complaint along the lines of "Could not find rake-0.8.7 in any of the sources" or similar. In the specific case of the Rake gem, Bundler would normally use the system version of the gem (at /Library/Ruby/Gems/1.8/gems/rake-0.8.7 on my machine) but when running inside JRuby it can't "see" any such system gems and can only "see" system gems under "/usr/local/jruby/lib/ruby/gems/1.8/gems". BUNDLE_PATH is set in the child process but that doesn't help in this case because the Rake gem is elsewhere.

Of course, I can install that gem using "sudo jruby -S gem install rake", but then it will just complain about another gem from the Gemfile. And I can't just run Bundler under JRuby (via "jruby -S bundle install") because it will choke on any gems which involve native C extensions with no Java variant available.

The workaround, for now, then, is to prune the RUBYOPT environment variable at the bottom of my "spec/spec_helper.rb", thus allowing JRuby to run without evaluating "bundler/setup" and without choking. The load path is already propagated to the JRuby process anyway, so in my specific case, the requires that JRuby does (most importantly, require 'celerity') will work and pull in the Bundler-managed version of that gem.

if ENV['RUBYOPT']
  ENV['RUBYOPT'] = ENV['RUBYOPT'].gsub(%r{-r\s*bundler/setup}, '')
end

So, not really sure what the right thing to do here is. The basic problem is that we have a Gemfile that specifies the requirements for an app running under MRI, and when that app ends up forking a child process that runs on another Ruby platform, we're going to run into problems unless that platform satisfies the same requirements, which is something that may not always be possible, or even make much sense. In the case of the example here, the JRuby process just wraps HtmlUnit and doesn't actually need any other gems at all (no dependencies, all the "requires" it does come from the standard library).

Like I said, don't really know what the solution is here. Is adding "-rbundler/setup" to RUBYOPT really the right thing to do (as far as I can tell, the load path will already be set up, so the only important thing Bundler.setup is doing is crippling Rubygems)? Is there a way Bundler.setup could be changed to avoid this breakage? Or do we need a way of specifying in a Gemfile that certain groups or gems should not be considered under certain platforms? Is my hacky workaround of pruning RUBYOPT in my spec/spec_helper.rb going to be the best solution?

indirect commented 14 years ago

FWIW, the Gemfile supports specifying that certain groups of gems should only be loaded under certain platforms as of version 1.0b2. You can test the beta with gem install bundler --pre. Here's a simple Gemfile example: http://gist.github.com/444065

jeremy commented 14 years ago

The problem isn't platform support within the bundle, it's that the bundle extends its grasp to programs outside the bundle via RUBYOPT, if those programs happen to be written in Ruby.

wincent commented 14 years ago

Having a play with 1.0b2 now. Looks like it's got to be "platforms" though, not "platform" like in that Gist:

src/Gemfile:5:in `build': undefined method `platform' for #<Bundler::Dsl:0x10106c4d8> (NoMethodError)
    from /Library/Ruby/Gems/1.8/gems/bundler-1.0.0.beta.2/lib/bundler.rb:128:in `definition'

Aliasing platform to platforms might not be a bad idea.

While this is definitely a nice addition, there are still problems with it under a mixed environment. For example, if I put most of my gems inside a "platforms :ruby_18" block and some others inside a "platforms :jruby" block then when I run "bundle" from the command line under Ruby 1.8 it will delete the JRuby gems from the vendor/cache:

Updating .gem files in vendor/cache
Removing outdated .gem files from vendor/cache
  * celerity-0.7.9.gem
  * jruby-openssl-0.7.gem

This isn't what I want, for obvious reasons. All the gems required by the project should be included in vendor/cache.

So, another thing I experimented with in order to prevent this unwanted deletion from the vendor/cache was to enclose the entire Gemfile inside a "platforms :ruby_18" block, including lines for Celerity etc:

# for JRuby: just get them installed and in the load path, but don't require them
gem 'celerity',       :require => nil
gem 'jruby-openssl',  :require => nil

(ie. I had no "platforms :jruby" block at all).

This does stop the Gems from getting deleted from vendor/cache, but then my spec suite blows up when it tries to fork a JRuby child process to run Celerity because RUBYOPT is still set up to execute "bundler/setup", and it explodes because it clears the load path and the needed gems can't be found.

So finally, this is what I have in my Gemfile and it seems to work for now:

platforms :ruby_18 do
  # all my standard gems
  gem 'haml'
  gem 'mysql'
  ...

  # gems needed within JRuby environment
  gem 'celerity', :require => nil
  gem 'jruby-openssl', :require => nil
end

platforms :jruby do
  gem 'celerity', :require => nil
  gem 'jruby-openssl', :require => nil
end

Looks like the gems needed by JRuby are all lumped in with the other gems inside BUNDLE_PATH, and when the child process is forked, even if the load path is cleared when "bundler/setup" is invoked via RUBYOPT, it looks like it adds back the right paths and Celerity seems to be working for now.

Cheers, Wincent

wincent commented 14 years ago

Ah, one more thing that caught me off guard:

platforms :ruby_18 do
  gem 'celerity', :require => nil
  gem 'jruby-openssl', :require => nil
end

platforms :jruby do
  gem 'celerity'
  gem 'jruby-openssl'
end

If I drop the ":require => nil" off those lines in the "platforms :jruby" block then even when running under Ruby 1.8.7 Bundler will try to require them and fail:

(in /Users/wincent/trabajo/unversioned/wincent.com/src)
rake aborted!
Celerity only works on JRuby at the moment.

So looks like that might be a bug. When running under MRI it shouldn't even know about what's inside the "jruby" block.

Cheers, Wincent

rubygems / bundler

Trouble in mixed MRI/JRuby environment #407