Hydra - "rake test" hanging

[christopherdolan]

I'm undertaking an effort to make Hydra more compatible with Cucumber 0.8.5 (latest). I've forked the project on GitHub, and I'm making my changes directly to that fork.

After my first change, I realized I should be running the tests you've provided! Unfortunately, they hang when I run them, even on your HEAD revision. I wanted to verify with you that this is unexpected behavior. After letting the tests run for a few minutes, I terminated them manually and got the callstack below.

$:~/proj/foss/hydra$ rake test (in ~/proj/foss/hydra) ~/.rvm/gems/ruby-1.8.7-p299@hydra-dev/gems/jeweler-1.4.0/lib/jeweler/commands/check_dependencies.rb:13:Warning: Gem::Dependency#version_requirements is deprecated and will be removed on or after August 2010. Use #requirement All dependencies seem to be installed. ~/.rvm/rubies/ruby-1.8.7-p299/bin/ruby -I"lib:lib:test" "~/.rvm/gems/ruby-1.8.7-p299@global/gems/rake-0.8.7/lib/rake/rake_test_loader.rb" "test/sync_test.rb" "test/runner_test.rb" "test/pipe_test.rb" "test/fixtures/sync_test.rb" "test/worker_test.rb" "test/message_test.rb" "test/master_test.rb" "test/ssh_test.rb" Loaded suite ~/.rvm/gems/ruby-1.8.7-p299@global/gems/rake-0.8.7/lib/rake/rake_test_loader Started .....^C./test/../lib/hydra/master.rb:216:in join': Interrupt from ./test/../lib/hydra/master.rb:216:inprocess_messages' from ./test/../lib/hydra/master.rb:216:in each' from ./test/../lib/hydra/master.rb:216:inprocess_messages' from ./test/../lib/hydra/master.rb:86:in initialize' from ./test/master_test.rb:84:innew' from ./test/master_test.rb:84:in __bind_1282173670_488554' from ~/.rvm/gems/ruby-1.8.7-p299@hydra-dev/gems/shoulda-2.10.3/lib/shoulda/context.rb:362:incall' from ~/.rvm/gems/ruby-1.8.7-p299@hydra-dev/gems/shoulda-2.10.3/lib/shoulda/context.rb:362:in test: with a file to test and a destination to verify should run a test via ssh. ' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testcase.rb:78:insend' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testcase.rb:78:in run' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:34:inrun' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:33:in each' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:33:inrun' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:34:in run' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:33:ineach' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:33:in run' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/ui/testrunnermediator.rb:46:inrun_suite' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/ui/console/testrunner.rb:67:in start_mediator' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/ui/console/testrunner.rb:41:instart' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/ui/testrunnerutilities.rb:29:in run' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/autorunner.rb:216:inrun' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/autorunner.rb:12:in `run' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit.rb:279 from ~/.rvm/gems/ruby-1.8.7-p299@global/gems/rake-0.8.7/lib/rake/rake_test_loader.rb:5 rake aborted!

[ngauthier]

The tests should definitely pass. You will need this: http://wiki.github.com/ngauthier/hydra/development-environment

-Nick

[christopherdolan]

I can SSH into my own machine via "ssh localhost" without entering a password. Below are my installed gems. How long should these tests take? The process seems to hang on the 6th test.

LOCAL GEMS: builder (2.1.2) cucumber (0.7.2) diff-lcs (1.1.2) gemcutter (0.6.1) gherkin (2.1.5) git (1.2.5) jeweler (1.4.0) json_pure (1.4.6) rake (0.8.7) rspec (1.3.0) rubyforge (2.0.4) shoulda (2.10.3) term-ansicolor (1.0.5) therubyracer (0.7.4) trollop (1.16.2)

[ngauthier]

Tests should take under a minute.

Install the gem "therubyracer". It is a development dependency and I think I forgot to put it in the Rakefile. Then try again.

If that doesn't work, can you run the test files individually and tell me which one is failing?

[christopherdolan]

As you can see from the list above, I have that gem installed already. It looks like the hanging test is: should "run a test via ssh" in test/master_test.rb:83

[christopherdolan]

looks like it's hanging on master.rb:216, waiting for all the listeners to become joinable

[ngauthier]

ah my bad I missed it, sorry.

Are you on a mac? I think some people on macs can't run "ssh -tt" and have the remote workers still work. On linux this is the preferred behavior.

Check this line out: http://github.com/ngauthier/hydra/blob/master/lib/hydra/ssh.rb#L29

Try removing the -tt and see if that helps.

Please don't commit the removal of the -tt though. Most people use linux servers for their remote workers.

-Nick

[christopherdolan]

Actually, I'm running this on Ubuntu Lucid Lynx x64. I'll give it a shot anyway.

[christopherdolan]

No luck. I first tried shelling into my own machine w/ "ssh -tt localhost". That worked fine. I added "verbose => true" to the initialization options for the Hydra::Master object, and here's the output I got when I re-ran the test:

Loaded suite test/master_test Started 1282233599.7289 MASTER| Initialized 1282233599.72903 MASTER| Files: (["~/proj/foss/hydra/test/fixtures/write_file.rb"]) 1282233599.72906 MASTER| Workers: ([{:type=>:ssh, :runners=>1, :connect=>"localhost", :directory=>"/home/cdolan/proj/foss/hydra/lib"}]) 1282233599.72907 MASTER| Verbose: (true) 1282233599.72909 MASTER| Booting 1 workers 1282233599.72911 MASTER| worker opts {"directory"=>"~/proj/foss/hydra/lib", "connect"=>"localhost", "type"=>:ssh, "runners"=>1} 1282233599.72913 MASTER| Processing Messages 1282233599.72915 MASTER| Workers: [{"directory"=>"~/proj/foss/hydra/lib", "connect"=>"localhost", "type"=>:ssh, "runners"=>1}] 1282233599.72931 MASTER| Listening to {"directory"=>"~/proj/foss/hydra/lib", "connect"=>"localhost", "type"=>:ssh, "runners"=>1} 1282233599.72943 MASTER| Booting SSH worker

this is where it hangs, even with the -tt option removed from the source, as you suggested.

[ngauthier]

that's odd. It will spit out a lot more if it is able to make the ssh connection.

Do you have the hydra gem installed? The "remote worker" (which is really local) will try to require hydra.

-Nick

[christopherdolan]

I think you hit on something there. I use RVM, and hydra is not in my system default gemset. I added an .rvmrc file pointing to my desired Ruby version & gemset to the project root directory, and now the test completes!

[ngauthier]

cool!

Yeah I am in the same boat, except I was doing hydra in my default ruby and gemset.

Hydra really needs to have better errors for remote workers. Right now it's really basic.

[christopherdolan]

I'm not sure it's appropriate that the tests require the gem to be built and installed. That said, a better solution does not immediately come to mind.

[ngauthier]

I agree on both counts. The remoting functionality of hydra is pretty immature, mainly because I don't use it on a day to day basis.

Weplay has been working on some improvements to hydra's remote workers:

http://github.com/weplay/hydra

I plan on pulling in many of their changes when I get some time.

-Nick

[christopherdolan]

Hate to resurrect the thread, but now the tests are hanging on "with a file to test and a destination to verify should synchronize a test file over ssh with rsync". Callstack below.

~/.rvm/gems/ruby-1.8.7-p299@hydra-dev/gems/jeweler-1.4.0/lib/jeweler/commands/check_dependencies.rb:13:Warning: Gem::Dependency#version_requirements is deprecated and will be removed on or after August 2010. Use #requirement All dependencies seem to be installed. ~/.rvm/rubies/ruby-1.8.7-p299/bin/ruby -I"lib:lib:test" "~/.rvm/gems/ruby-1.8.7-p299@global/gems/rake-0.8.7/lib/rake/rake_test_loader.rb" "test/sync_test.rb" "test/runner_test.rb" "test/pipe_test.rb" "test/fixtures/sync_test.rb" "test/worker_test.rb" "test/message_test.rb" "test/master_test.rb" "test/ssh_test.rb" Loaded suite ~/.rvm/gems/ruby-1.8.7-p299@global/gems/rake-0.8.7/lib/rake/rake_test_loader Started ........^C./test/../lib/hydra/master.rb:216:in join': Interrupt from ./test/../lib/hydra/master.rb:216:inprocess_messages' from ./test/../lib/hydra/master.rb:216:in each' from ./test/../lib/hydra/master.rb:216:inprocess_messages' from ./test/../lib/hydra/master.rb:86:in initialize' from ./test/master_test.rb:131:innew' from ./test/master_test.rb:131:in __bind_1282317433_132319' from ~/.rvm/gems/ruby-1.8.7-p299@hydra-dev/gems/shoulda-2.10.3/lib/shoulda/context.rb:362:incall' from ~/.rvm/gems/ruby-1.8.7-p299@hydra-dev/gems/shoulda-2.10.3/lib/shoulda/context.rb:362:in test: with a file to test and a destination to verify should synchronize a test file over ssh with rsync. ' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testcase.rb:78:insend' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testcase.rb:78:in run' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:34:inrun' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:33:in each' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:33:inrun' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:34:in run' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:33:ineach' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/testsuite.rb:33:in run' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/ui/testrunnermediator.rb:46:inrun_suite' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/ui/console/testrunner.rb:67:in start_mediator' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/ui/console/testrunner.rb:41:instart' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/ui/testrunnerutilities.rb:29:in run' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/autorunner.rb:216:inrun' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit/autorunner.rb:12:in `run' from ~/.rvm/rubies/ruby-1.8.7-p299/lib/ruby/1.8/test/unit.rb:279 from ~/.rvm/gems/ruby-1.8.7-p299@global/gems/rake-0.8.7/lib/rake/rake_test_loader.rb:5 rake aborted! Command failed with status (): [~/.rvm/rubies/ruby-1.8.7-p299/b...]

[ngauthier]

Can you post the verbose mode output? You can use gist.github.com to post traces and code and stuff. Cleaner than in here.

[christopherdolan]

here ya go: http://gist.github.com/540546

[ngauthier]

so the sync worked, but then the ssh connection died just like last time.

Your rvm settings didn't change or anything? Do you need to re-install the gem after making changes?

[christopherdolan]

No, the RVM settings haven't changed. However, the '.rvmrc' file is only read when the user returns to the hydra project folder. If the worker never passes through that folder, the global gemset is used, which doesn't include hydra. I wonder if it's worthwhile to validate that the target environment is configured properly, and fast-fail if it isn't...

[ngauthier]

Yeah. That's what I plan on doing. Hydra needs to return a hydra message reporting an error if hydra isn't available. Right now it just hangs like that.

If you install hydra to your global gemset you should probably be ok.

[ngauthier]

You are one lucky dude. I just so happened to need cucumber 0.8.5 support today on a project.

I actually got the same issue as you running the test suite. It was because of rvm. Hydra works in a tmp directory for its tests, so the only gemset available is the default one and global. So I did:

rvm gemset use global
rake install

Then the test suite ran.

Cucumber 0.8.5 was pretty easy, just a slight method change.

Hydra 0.20.0 is out w/ cuke 0.8.5!

[PeteTheSadPanda]

So I've run into the same issue, I'm using rvm on a mac, and my tests are haning after the cuke warnings that the specs say can be ignored. Do the various spec dependencies need to be in the global gemset as well? I got stuck 6 specs in, but then read this thread and got much further, still am seeing 2 failures and hanging like I said three specs after the warnings. This is for 0.20.0, I'd prefer not to have to pollute my global gemset, but it's not the end of the world. That's what rvm gemset empty is for :-D.

[undees]

I can recreate this on Lion with rbenv (no gemsets) with the latest Cucumber.