search

grosser / parallel_tests

Ruby: 2 CPUs = 2x Testing Speed for RSpec, Test::Unit and Cucumber
3.38k stars 494 forks source link

Jruby JVM options #256

Closed westlakem closed 10 years ago

westlakem commented 11 years ago

Working for a large company, we are using the parallel_tests gem to run our cucumber automation. This works well for our Ruby devisions, but for our Jruby folk we need another option. The issue is that Jruby fork spawns a new java JVM without the JVM options. I have figured out that I need 1 of 2 solutions and I'm not exactly sure how to implement either of them.

  1. We need to be able to pass the JVM options into fork so we can get all the options we need.

or

  1. We need to change how the gem handles forking so the processes all run on the same JVM

I don't know if either of these solutions are a possibility, but maybe someone else will know better

According to a solution posted at http://stackoverflow.com/questions/11818463/alternative-for-spawning-a-process-with-fork-in-jruby

I found out the solution for this. We can use the built-in library FFI in JRuby to 'simulate' the Process.fork in MRI.

To mimic the Process.fork in MRI Ruby

module JRubyProcess require 'ffi' extend FFI::Library ffi_lib FFI::Library::LIBC attach_function :fork, [], :int end

pid = JRubyProcess.fork do

do_your_work

end More details:

https://github.com/ffi/ffi

http://blog.headius.com/2008/10/ffi-for-ruby-now-available.html

another option was to use spoon: https://github.com/headius/spoon

I am no linux guru, but these seem like simple ways to implement Jruby. Does anyone know if this is an effective solution?

grosser commented 11 years ago

Not sure if I understand the problem, so normally you run the tests via ruby xxx_test.rb yy_test.rb but if parallel_tests executes a subshell that does ruby xxx_test.rb then you are missing jsm options ?

How do you normally pass those jvm options ?

JVM_FOO=1 ruby xxx ?

westlakem commented 11 years ago

So after doing some digging through JRuby 1.7.4, how JRuby handles forking is that it spawns a new JVM with the default options.

When I execute Jruby from the command line, I can execute it like the following:

jruby -J-Xms128m -J-Xmx1024m -S parallel_cucumber features/A -n 4 -o 'TEST_ENV=localhost'

This causes the main parent JVM to start with those 2 options (the -J signifies a java option to Jruby). When the process gets forked, Jruby spawns a new JVM, and since the java options are not part of the JVM call, they just get loaded with the system defaults.

Since this is done by Jruby, it is impossible to pass in JVM args to the new processes without modifying JRuby. Is it possible to use forking with Ruby / Ruby MRI, and Java threading for JRuby? Once again, i'm not sure how Forking works, so I really don't know what it would take for that to work.

Hope that answered your question

grosser commented 11 years ago

Parallel_tests uses subshells to run the tests, they cannot be run in threads, so if this is to work we somehow have to pass the jruby options to the sub-shells, would passing them via parallel_rpsec --test-options '-J-xyz' work ?

On Wed, Sep 11, 2013 at 8:30 PM, nightbeast notifications@github.comwrote:

So after doing some digging through JRuby 1.7.4, how JRuby handles forking is that it spawns a new JVM with the default options.

When I execute Jruby from the command line, I can execute it like the following:

jruby -J-Xms128m -J-Xmx1024m -S parallel_cucumber features/A -n 4 -o 'TEST_ENV=localhost'

This causes the main parent JVM to start with those 2 options (the -J signifies a java option to Jruby). When the process gets forked, Jruby spawns a new JVM, and since the java options are not part of the JVM call, they just get loaded with the system defaults.

Since this is done by Jruby, it is impossible to pass in JVM args to the new processes without modifying JRuby. Is it possible to use forking with Ruby / Ruby MRI, and Java threading for JRuby? Once again, i'm not sure how Forking works, so I really don't know what it would take for that to work.

Hope that answered your question

— Reply to this email directly or view it on GitHubhttps://github.com/grosser/parallel_tests/issues/256#issuecomment-24292988 .

westlakem commented 11 years ago

In short... no... --test-options (or -o) pass args into the test command. This causes: jruby - S parallel_cucumber -o '-J-xyz' to act as jruby -S cucumber -J-xyz we need to get the -J infront of the -S in the cucumber call: jruby -J-xyz -S cucumber

grosser commented 11 years ago

so something like --prefix-command 'jruby -XXX -S' that then turns cucumber into jruby -XXX -S cucmber ? any chance we can detect the current options and just do something like when on jruby inherit current options ?

On Thu, Sep 12, 2013 at 10:32 AM, nightbeast notifications@github.comwrote:

In short... no... --test-options (or -o) pass args into the test command. This causes: jruby - S parallel_cucumber -o '-J-xyz' to act as jruby -S cucumber -J-xyz we need to get the -J infront of the -S in the cucumber call: jruby -J-xyz -S cucumber

— Reply to this email directly or view it on GitHubhttps://github.com/grosser/parallel_tests/issues/256#issuecomment-24340103 .

westlakem commented 11 years ago

That would work if we have access to the command that is being executed in the sub shells. Do you know where exactly in the gem that call is being made? If so I could look at it and possibly show you what is needed.

grosser commented 11 years ago

I think lib/parallel_tests/test/runner.rb:52 is what you are looking for

On Thu, Sep 12, 2013 at 10:46 AM, nightbeast notifications@github.comwrote:

That would work if we have access to the command that is being executed in the sub shells. Do you know where exactly in the gem that call is being made? If so I could look at it and possibly show you what is needed.

— Reply to this email directly or view it on GitHubhttps://github.com/grosser/parallel_tests/issues/256#issuecomment-24341170 .

westlakem commented 10 years ago

Ok, I think I see something we can use. Although I cannot set system environment variables, I can set shell specific environment variables. Line 69 handles export commands that are generated from the env.map passed into the execute_command_and_capture_output method at line 58 by way of the env parameter.

see https://github.com/grosser/parallel_tests/pull/258

westlakem commented 10 years ago

Mike, Look at https://github.com/grosser/parallel_tests/pull/259

From: Michael Grosser notifications@github.com To: grosser/parallel_tests parallel_tests@noreply.github.com Cc: nightbeast westlake_m@Yahoo.com Sent: Thursday, September 12, 2013 1:58 PM Subject: Re: [parallel_tests] Jruby JVM options (#256)

I think lib/parallel_tests/test/runner.rb:52 is what you are looking for

On Thu, Sep 12, 2013 at 10:46 AM, nightbeast notifications@github.comwrote:

That would work if we have access to the command that is being executed in the sub shells. Do you know where exactly in the gem that call is being made? If so I could look at it and possibly show you what is needed.

— Reply to this email directly or view it on GitHubhttps://github.com/grosser/parallel_tests/issues/256#issuecomment-24341170 .

— Reply to this email directly or view it on GitHub.