search

jasmine / jasmine-gem

Jasmine ruby gem
682 stars 275 forks source link

jasmine:ci rake task doesn't return non zero exit code if exception occurs outside of tests #131

Closed lyahdav closed 11 years ago

lyahdav commented 11 years ago

I had a situation where our CI server ran the jasmine:ci rake task and we got output like this:

>> Thin web server (v1.5.0 codename Knife)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:55535, CTRL+C to stop
Waiting for jasmine server on 55535...
jasmine server started.
Waiting for suite to finish in browser ...
/Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/protocol.rb:146:in `rescue in rbuf_fill': Timeout::Error (Timeout::Error)
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/protocol.rb:140:in `rbuf_fill'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/http.rb:1293:in `request'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in `request'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/newrelic_rpm-3.5.6.42.beta/lib/new_relic/agent/instrumentation/net.rb:22:in `block in request_with_newrelic_trace'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/newrelic_rpm-3.5.6.42.beta/lib/new_relic/agent/method_tracer.rb:240:in `trace_execution_scoped'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/newrelic_rpm-3.5.6.42.beta/lib/new_relic/agent/instrumentation/net.rb:21:in `request_with_newrelic_trace'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/http.rb:1286:in `block in request'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/http.rb:745:in `start'
    from /Users/Admin/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/net/http.rb:1284:in `request'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in `request'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/newrelic_rpm-3.5.6.42.beta/lib/new_relic/agent/instrumentation/net.rb:22:in `block in request_with_newrelic_trace'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/newrelic_rpm-3.5.6.42.beta/lib/new_relic/agent/method_tracer.rb:240:in `trace_execution_scoped'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/newrelic_rpm-3.5.6.42.beta/lib/new_relic/agent/instrumentation/net.rb:21:in `request_with_newrelic_trace'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/selenium-webdriver-2.21.2/lib/selenium/webdriver/remote/http/default.rb:81:in `response_for'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/selenium-webdriver-2.21.2/lib/selenium/webdriver/remote/http/default.rb:43:in `request'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/selenium-webdriver-2.21.2/lib/selenium/webdriver/remote/http/common.rb:40:in `call'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/selenium-webdriver-2.21.2/lib/selenium/webdriver/remote/bridge.rb:598:in `raw_execute'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/selenium-webdriver-2.21.2/lib/selenium/webdriver/remote/bridge.rb:576:in `execute'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/selenium-webdriver-2.21.2/lib/selenium/webdriver/remote/bridge.rb:322:in `executeScript'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/selenium-webdriver-2.21.2/lib/selenium/webdriver/common/driver.rb:211:in `execute_script'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/jasmine-1.2.1/lib/jasmine/selenium_driver.rb:33:in `eval_js'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/jasmine-1.2.1/lib/jasmine/runners/http.rb:61:in `eval_js'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/jasmine-1.2.1/lib/jasmine/runners/http.rb:55:in `wait_for_suites_to_finish_running'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/jasmine-1.2.1/lib/jasmine/runners/http.rb:14:in `run'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/jasmine-1.2.1/lib/jasmine/run_specs.rb:30:in `<top (required)>'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/activesupport-3.2.11/lib/active_support/dependencies.rb:245:in `load'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/activesupport-3.2.11/lib/active_support/dependencies.rb:245:in `block in load'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/activesupport-3.2.11/lib/active_support/dependencies.rb:236:in `load_dependency'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/activesupport-3.2.11/lib/active_support/dependencies.rb:245:in `load'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/rspec-core-2.12.2/lib/rspec/core/configuration.rb:789:in `block in load_spec_files'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/rspec-core-2.12.2/lib/rspec/core/configuration.rb:789:in `each'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/rspec-core-2.12.2/lib/rspec/core/configuration.rb:789:in `load_spec_files'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/rspec-core-2.12.2/lib/rspec/core/command_line.rb:22:in `run'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/rspec-core-2.12.2/lib/rspec/core/runner.rb:77:in `rescue in run'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/rspec-core-2.12.2/lib/rspec/core/runner.rb:73:in `run'
    from /Users/Admin/.rvm/gems/ruby-1.9.3-p327@handl-server/gems/rspec-core-2.12.2/lib/rspec/core/runner.rb:17:in `block in autorun'

As you can see, there was a Timeout::Error for some reason outside of the test runs themselves, but this process ended with an exit status of 0. This caused our CI process to think the jasmine tests succeeded when they in fact didn't.

ptolemarch commented 11 years ago

The same thing happens right now if you're running Firefox Beta (Firefox 19), which doesn't seem to be supported by Selenium just yet:

/Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/selenium-webdriver-2.29.0/lib/selenium/webdriver/firefox/launcher.rb:79:in `connect_until_stable': unable to obtain stable firefox connection in 60 seconds (127.0.0.1:7055) (Selenium::WebDriver::Error::WebDriverError)
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/selenium-webdriver-2.29.0/lib/selenium/webdriver/firefox/launcher.rb:37:in `block in launch'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/selenium-webdriver-2.29.0/lib/selenium/webdriver/firefox/socket_lock.rb:20:in `locked'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/selenium-webdriver-2.29.0/lib/selenium/webdriver/firefox/launcher.rb:32:in `launch'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/selenium-webdriver-2.29.0/lib/selenium/webdriver/firefox/bridge.rb:20:in `initialize'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/selenium-webdriver-2.29.0/lib/selenium/webdriver/common/driver.rb:31:in `new'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/selenium-webdriver-2.29.0/lib/selenium/webdriver/common/driver.rb:31:in `for'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/selenium-webdriver-2.29.0/lib/selenium/webdriver.rb:67:in `for'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/bundler/gems/jasmine-gem-479dccb62f9e/lib/jasmine/selenium_driver.rb:20:in `initialize'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/bundler/gems/jasmine-gem-479dccb62f9e/lib/jasmine/run_specs.rb:20:in `new'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/bundler/gems/jasmine-gem-479dccb62f9e/lib/jasmine/run_specs.rb:20:in `<top (required)>'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `load'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `block in load'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:236:in `load_dependency'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `load'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/rspec-core-2.10.1/lib/rspec/core/configuration.rb:746:in `block in load_spec_files'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/rspec-core-2.10.1/lib/rspec/core/configuration.rb:746:in `map'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/rspec-core-2.10.1/lib/rspec/core/configuration.rb:746:in `load_spec_files'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/rspec-core-2.10.1/lib/rspec/core/command_line.rb:22:in `run'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/rspec-core-2.10.1/lib/rspec/core/runner.rb:69:in `run'
    from /Users/cogent/.rvm/gems/ruby-1.9.3-p194@slashmob/gems/rspec-core-2.10.1/lib/rspec/core/runner.rb:10:in `block in autorun'

Since this still returns as a success, rake ship proceeds unimpeded.

cantino commented 11 years ago

Does anyone know why this is not returning a non-zero exit status?

mkornblum commented 11 years ago

We just ran into this as well, Jasmine CI task fails but still returns 0, so the build proceeds as if all is well.

cantino commented 11 years ago

We never figured out why this was happening. It'd be great to understand the root cause.

jboyens commented 11 years ago

This is likely due to the fact that we don't wrap the entire run script in a begin...rescue. I've created a feature in Tracker for this: https://www.pivotaltracker.com/story/show/54066951