search

sauce-archives / sauce_ruby

This is the Ruby client adapter for testing with Sauce Labs, a Selenium-based browser testing service (saucelabs.com).
Apache License 2.0
98 stars 115 forks source link

Tests always return 0 exit status regardless of result #139

Open chrislloyd opened 11 years ago

chrislloyd commented 11 years ago

Here's what my test suite looks like without the Sauce gem required:

# In Gemfile
gem 'rspec-rails'
gem 'capybara'

$ rspec
# ... (tests fail)
$ echo $?
1

Then with the Sauce gem added (and no extra configuration):

# In Gemfile
gem 'rspec-rails'
gem 'capybara'
gem 'sauce', github: 'sauce-labs/sauce_ruby'

$ rspec
# ... (tests still fail)
$ echo $?
0

This kind of breaks everything, but most importantly it breaks most CI.

DanielHeath commented 11 years ago

Yep, seeing this too.

Pretty much a deal breaker.

DanielHeath commented 11 years ago

I suspect https://github.com/sauce-labs/sauce_ruby/blob/master/lib/sauce/capybara.rb#L104 is causing it, will investigate.

DanielHeath commented 11 years ago

Confirmed! Took that line out, all working. PR will come later.

DylanLacey commented 11 years ago

This should be fixed in v2.4.6.

Ultimately, the root cause was a bug in Ruby, http://bugs.ruby-lang.org/issues/5218. This is fixed for Ruby 1.9.2-p392 but lower versions still have the issue.

DanielHeath commented 11 years ago

This has broken again (between 1553347fdc24682e592a51796ed5b76f0223f08f and d9d5cb7a65851849f09d5407eae0f7ccab9941df ).

Hard to add tests for but it might be worth finding a way.

DylanLacey commented 11 years ago

Hmm, I'll re-investigate.

I think there's a case for re-architecting all the current examples and tests into a set of standalone integration tests, then kicking them off in separate processes and checking for key outputs using IO objects and the like.

DylanLacey commented 11 years ago

I'm having problems replicating this, even between those commits.

Does the current release version of the gem have the same problem? Can you attach a Gemfile.lock so I can try to replicate myself?

DanielHeath commented 11 years ago

Gemfile*: https://gist.github.com/DanielHeath/5753478 Testing with latest master now.

DanielHeath commented 11 years ago

Still (not) failing with latest master. Updated the gist with the new gemfile.

DanielHeath commented 11 years ago

Some extra context: I'm running rake sauce:spec with a single browser configured. Doesn't matter which one I use (tried linux/win/osx with firefox/IE/chrome)

DylanLacey commented 11 years ago

I think this has been fixed with 89abd334e38c4b55d15ecb348f6cfa970d2e1882, which has been released as 3.0.1. I was unable to repro for rspec by itself still.