search

defunctzombie / zuul

[UNMAINTAINED] multi-framework javascript browser testing
956 stars 87 forks source link

Getting Error: Not running when running tests against SauceLabs #129

Closed dagingaa closed 9 years ago

dagingaa commented 9 years ago

I'm writing some tests that I want running on SauceLabs through zuul. It works fine locally, but when running against SauceLabs it fails at different points in the process, sometimes completing (1 out of 10 runs, if not less), and sometimes failing with the following error. I see no pattern, other than that it is the same error every time, but it happens at different times in the run.

➜  appearin-sdk git:(zuul) ✗ npm test

> appearin-sdk@0.0.3 test /Users/dagingaa/code/comoyo/appearin-sdk
> zuul -- test.js

- testing: chrome @ Windows 2003: 30
- testing: chrome @ Mac 10.6: 31 33 37
- testing: chrome @ Windows 2012 R2: 32 34
- testing: chrome @ Mac 10.9: 35
- testing: chrome @ Mac 10.8: 36 39
- testing: chrome @ Windows 2012: 38
- testing: firefox @ Windows 2008: 27 32
- testing: firefox @ Linux: 28
- testing: firefox @ Mac 10.6: 29
- testing: firefox @ Mac 10.9: 30
- testing: firefox @ Windows 2003: 31
- testing: firefox @ Windows 2012: 33 35
- testing: firefox @ Windows 2012 R2: 34
- testing: opera @ Windows 2003: 12
- testing: internet explorer @ Windows 2012: 10
- testing: internet explorer @ Windows 2012 R2: 11
- testing: safari @ Mac 10.8: 6
- testing: safari @ Mac 10.9: 7
- testing: safari @ Mac 10.10: 8
- queuing: <chrome 32 on Windows 2012 R2>
- queuing: <chrome 30 on Windows 2003>
- queuing: <chrome 31 on Mac 10.6>
- starting: <chrome 30 on Windows 2003>
- starting: <chrome 32 on Windows 2012 R2>
- passed: <chrome 30 on Windows 2003>
- queuing: <chrome 33 on Mac 10.6>
- passed: <chrome 32 on Windows 2012 R2>
- queuing: <chrome 34 on Windows 2012 R2>
- starting: <chrome 31 on Mac 10.6>
- starting: <chrome 34 on Windows 2012 R2>
- passed: <chrome 31 on Mac 10.6>
- queuing: <chrome 35 on Mac 10.9>
- passed: <chrome 34 on Windows 2012 R2>
- queuing: <chrome 36 on Mac 10.8>
- starting: <chrome 35 on Mac 10.9>
- starting: <chrome 36 on Mac 10.8>
- passed: <chrome 35 on Mac 10.9>
- queuing: <chrome 37 on Mac 10.6>
- passed: <chrome 36 on Mac 10.8>
- queuing: <chrome 38 on Windows 2012>
- starting: <chrome 38 on Windows 2012>
- starting: <chrome 37 on Mac 10.6>
- passed: <chrome 38 on Windows 2012>
- queuing: <chrome 39 on Mac 10.8>
- passed: <chrome 37 on Mac 10.6>
- queuing: <firefox 27 on Windows 2008>
- starting: <chrome 39 on Mac 10.8>
- starting: <firefox 27 on Windows 2008>
- passed: <chrome 39 on Mac 10.8>
- queuing: <firefox 28 on Linux>
- starting: <firefox 28 on Linux>
- passed: <firefox 27 on Windows 2008>
- queuing: <firefox 29 on Mac 10.6>
- starting: <chrome 33 on Mac 10.6>
- passed: <firefox 28 on Linux>
- queuing: <firefox 30 on Mac 10.9>
- starting: <firefox 29 on Mac 10.6>
- passed: <chrome 33 on Mac 10.6>
- queuing: <firefox 31 on Windows 2003>
- starting: <firefox 30 on Mac 10.9>
- starting: <firefox 31 on Windows 2003>
- passed: <firefox 29 on Mac 10.6>
- queuing: <firefox 32 on Windows 2008>
- passed: <firefox 30 on Mac 10.9>
- queuing: <firefox 33 on Windows 2012>
- starting: <firefox 32 on Windows 2008>
- passed: <firefox 31 on Windows 2003>
- queuing: <firefox 34 on Windows 2012 R2>
- starting: <firefox 33 on Windows 2012>
- starting: <firefox 34 on Windows 2012 R2>
- passed: <firefox 33 on Windows 2012>

Error: Not running
    at Server.close (net.js:1236:11)
    at Object.shutdown (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/setup.js:78:17)
    at finish_shutdown (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:206:29)
    at /Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:235:13
    at cb (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:221:45)
    at null.<anonymous> (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/callbacks.js:23:11)
    at cb (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:160:22)
    at /Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:174:5
    at Request._callback (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/http-utils.js:85:7)
    at Request.self.callback (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:364:22)
    at Request.init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:367:10)
    at new Request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:282:8)
    at request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/index.js:50:10)
    at Object.requestWithRetry (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/http-utils.js:72:3)
    at Webdriver._jsonWireCall (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:171:13)
    at commands.quit (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/commands.js:483:8)
    at Webdriver.(anonymous function) [as quit] (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:225:15)
    at SauceBrowser.shutdown (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:229:22)
    at Request.init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:367:10)
    at new Request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:282:8)
    at request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/index.js:50:10)
    at Object.requestWithoutRetry (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/http-utils.js:92:3)
    at Webdriver._sauceJobUpdate (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:197:13)
    at commands.sauceJobStatus (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/commands.js:2332:24)
    at Webdriver.(anonymous function) [as sauceJobStatus] (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:225:15)
    at EventEmitter.<anonymous> (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:120:29)
    at /Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:116:26
    at cb (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:221:45)
    at /Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:125:17
    at Request._callback (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/http-utils.js:85:7)
    at Request.self.callback (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:364:22)
    at Request.EventEmitter.emit (events.js:98:17)
    at Request.<anonymous> (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:1318:14)
    at Request.init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:367:10)
    at new Request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:282:8)
    at request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/index.js:50:10)
    at Object.requestWithRetry (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/http-utils.js:72:3)
    at Webdriver._init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:103:13)
    at commands.init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/commands.js:33:14)
    at Webdriver.(anonymous function) [as init] (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:225:15)
    at init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:99:21)
npm ERR! Test failed.  See above for more details.
npm ERR! not ok code 0

Any ideas what's causing this, and can I add some configuration to simply catch and ignore that error? SauceLabs reports everything a-ok.

The tests I'm running can be found here: https://github.com/appearin/appearin-sdk/tree/zuul

dagingaa commented 9 years ago

Confirmed that this also happens when running on Travis-CI

dagingaa commented 9 years ago

I ran zuul with debug options enabled, and getting the following (same error, but more debug logs)

> DEBUG=zuul* zuul -- test.js

- testing: chrome @ Windows 2003: 30
- testing: chrome @ Mac 10.6: 31 33 35
- testing: chrome @ Linux: 32 34
- testing: chrome @ Mac 10.8: 36 37 39
- testing: chrome @ Windows 2012: 38
- testing: firefox @ Windows 2012: 27
- testing: firefox @ Windows 2008: 28 29 30
- testing: firefox @ Mac 10.6: 31
- testing: firefox @ Mac 10.9: 32
- testing: firefox @ Linux: 33 34 35
- testing: opera @ Windows 2003: 12
- testing: internet explorer @ Windows 2012: 10
- testing: internet explorer @ Windows 2012 R2: 11
- testing: safari @ Mac 10.8: 6
- testing: safari @ Mac 10.9: 7
- testing: safari @ Mac 10.10: 8
  zuul control server active on port 60667 +0ms
  zuul:saucebrowser running chrome 30 Windows 2003 +4ms
  zuul:saucebrowser running chrome 31 Mac 10.6 +4ms
  zuul:saucebrowser running chrome 32 Linux +1ms
  zuul:setup bouncer active on port 60668 +1ms
  zuul:setup bouncer active on port 60669 +2ms
  zuul:setup bouncer active on port 60670 +0ms
- queuing: <chrome 32 on Linux>
  zuul:saucebrowser queuing chrome 32 Linux +361ms
- queuing: <chrome 30 on Windows 2003>
  zuul:saucebrowser queuing chrome 30 Windows 2003 +9ms
- queuing: <chrome 31 on Mac 10.6>
  zuul:saucebrowser queuing chrome 31 Mac 10.6 +4ms
  zuul:saucebrowser open https://uuxsijtngm.localtunnel.me/__zuul +6s
- starting: <chrome 32 on Linux>
  zuul:saucebrowser open https://squzjnoszt.localtunnel.me/__zuul +175ms
- starting: <chrome 31 on Mac 10.6>
  zuul:saucebrowser open https://htzhrzsyqn.localtunnel.me/__zuul +32ms
- starting: <chrome 30 on Windows 2003>
  zuul:saucebrowser waiting for test results from https://squzjnoszt.localtunnel.me/__zuul +5s
  zuul:saucebrowser waiting for test results from https://uuxsijtngm.localtunnel.me/__zuul +4s
  zuul:saucebrowser res.length: 10 +409ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser res.length: 0 +287ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser waiting for test results from https://squzjnoszt.localtunnel.me/__zuul +716ms
  zuul:saucebrowser waiting for test results from https://uuxsijtngm.localtunnel.me/__zuul +285ms
  zuul:saucebrowser res.length: 10 +371ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser waiting for test results from https://squzjnoszt.localtunnel.me/__zuul +1s
  zuul:saucebrowser res.length: 10 +675ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser waiting for test results from https://htzhrzsyqn.localtunnel.me/__zuul +594ms
  zuul:saucebrowser waiting for test results from https://squzjnoszt.localtunnel.me/__zuul +410ms
  zuul:saucebrowser res.length: 10 +87ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser res.length: 0 +253ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser res.length: 8 +311ms
  zuul:saucebrowser done chrome 31 Mac 10.6 +0ms
  zuul:saucebrowser finished tests for https://squzjnoszt.localtunnel.me/__zuul +2ms
  zuul:saucebrowser waiting for test results from https://uuxsijtngm.localtunnel.me/__zuul +435ms
  zuul:saucebrowser waiting for test results from https://htzhrzsyqn.localtunnel.me/__zuul +255ms
  zuul:saucebrowser quitting browser +449ms
  zuul:saucebrowser res.length: 10 +12ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser shutdown +460ms
- passed: <chrome 31 on Mac 10.6>
  zuul:saucebrowser running chrome 33 Mac 10.6 +1ms
  zuul:setup bouncer active on port 60798 +2ms
- queuing: <chrome 33 on Mac 10.6>
  zuul:saucebrowser queuing chrome 33 Mac 10.6 +220ms
  zuul:saucebrowser waiting for test results from https://uuxsijtngm.localtunnel.me/__zuul +318ms
  zuul:saucebrowser res.length: 10 +733ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser res.length: 10 +460ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser waiting for test results from https://uuxsijtngm.localtunnel.me/__zuul +541ms
  zuul:saucebrowser waiting for test results from https://htzhrzsyqn.localtunnel.me/__zuul +464ms
  zuul:saucebrowser res.length: 8 +412ms
  zuul:saucebrowser done chrome 32 Linux +0ms
  zuul:saucebrowser finished tests for https://uuxsijtngm.localtunnel.me/__zuul +1ms
  zuul:saucebrowser res.length: 10 +297ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser waiting for test results from https://htzhrzsyqn.localtunnel.me/__zuul +1s
  zuul:saucebrowser quitting browser +602ms
  zuul:saucebrowser res.length: 10 +120ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser shutdown +466ms
- passed: <chrome 32 on Linux>
  zuul:saucebrowser running chrome 34 Linux +1ms
  zuul:setup bouncer active on port 60820 +1ms
- queuing: <chrome 34 on Linux>
  zuul:saucebrowser queuing chrome 34 Linux +205ms
  zuul:saucebrowser waiting for test results from https://htzhrzsyqn.localtunnel.me/__zuul +329ms
  zuul:saucebrowser res.length: 8 +734ms
  zuul:saucebrowser done chrome 30 Windows 2003 +0ms
  zuul:saucebrowser finished tests for https://htzhrzsyqn.localtunnel.me/__zuul +1ms
  zuul:saucebrowser quitting browser +2s
  zuul:saucebrowser shutdown +557ms
- passed: <chrome 30 on Windows 2003>
  zuul:saucebrowser running chrome 35 Mac 10.6 +1ms
  zuul:setup bouncer active on port 60837 +1ms
- queuing: <chrome 35 on Mac 10.6>
  zuul:saucebrowser queuing chrome 35 Mac 10.6 +208ms
  zuul:saucebrowser open https://vjejfjihqv.localtunnel.me/__zuul +3s
- starting: <chrome 33 on Mac 10.6>
  zuul:saucebrowser open https://nemmvhkvmb.localtunnel.me/__zuul +1s
- starting: <chrome 34 on Linux>
  zuul:saucebrowser waiting for test results from https://vjejfjihqv.localtunnel.me/__zuul +3s
  zuul:saucebrowser waiting for test results from https://nemmvhkvmb.localtunnel.me/__zuul +1s
  zuul:saucebrowser res.length: 10 +737ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser waiting for test results from https://nemmvhkvmb.localtunnel.me/__zuul +1s
  zuul:saucebrowser open https://vvjityeorm.localtunnel.me/__zuul +509ms
- starting: <chrome 35 on Mac 10.6>
  zuul:saucebrowser res.length: 10 +3s
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser waiting for test results from https://vjejfjihqv.localtunnel.me/__zuul +1s
  zuul:saucebrowser waiting for test results from https://vvjityeorm.localtunnel.me/__zuul +553ms
  zuul:saucebrowser res.length: 10 +165ms
  zuul:saucebrowser fetching more results +1ms
  zuul:saucebrowser res.length: 10 +9ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser res.length: 10 +973ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser waiting for test results from https://nemmvhkvmb.localtunnel.me/__zuul +18ms
  zuul:saucebrowser waiting for test results from https://vjejfjihqv.localtunnel.me/__zuul +12ms
  zuul:saucebrowser res.length: 10 +724ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser res.length: 10 +23ms
  zuul:saucebrowser fetching more results +0ms
  zuul:saucebrowser waiting for test results from https://vvjityeorm.localtunnel.me/__zuul +226ms
  zuul:saucebrowser waiting for test results from https://nemmvhkvmb.localtunnel.me/__zuul +756ms
  zuul:saucebrowser waiting for test results from https://vjejfjihqv.localtunnel.me/__zuul +19ms
  zuul:saucebrowser res.length: 8 +694ms
  zuul:saucebrowser done chrome 34 Linux +1ms
  zuul:saucebrowser finished tests for https://nemmvhkvmb.localtunnel.me/__zuul +1ms
  zuul:saucebrowser res.length: 8 +41ms
  zuul:saucebrowser done chrome 33 Mac 10.6 +0ms
  zuul:saucebrowser finished tests for https://vjejfjihqv.localtunnel.me/__zuul +1ms
  zuul:saucebrowser quitting browser +1s
  zuul:saucebrowser quitting browser +34ms
  zuul:saucebrowser shutdown +494ms
- passed: <chrome 33 on Mac 10.6>
  zuul:saucebrowser running chrome 36 Mac 10.8 +1ms
  zuul:setup bouncer active on port 60944 +2ms
  zuul:saucebrowser shutdown +48ms

Error: Not running
    at Server.close (net.js:1236:11)
    at Object.shutdown (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/setup.js:78:17)
    at finish_shutdown (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:206:29)
    at /Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:235:13
    at cb (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:221:45)
    at null.<anonymous> (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/callbacks.js:23:11)
    at cb (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:160:22)
    at /Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:174:5
    at Request._callback (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/http-utils.js:85:7)
    at Request.self.callback (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:364:22)
    at Request.init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:367:10)
    at new Request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:282:8)
    at request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/index.js:50:10)
    at Object.requestWithRetry (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/http-utils.js:72:3)
    at Webdriver._jsonWireCall (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:171:13)
    at commands.quit (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/commands.js:483:8)
    at Webdriver.(anonymous function) [as quit] (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:225:15)
    at SauceBrowser.shutdown (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:229:22)
    at Request.init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:367:10)
    at new Request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:282:8)
    at request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/index.js:50:10)
    at Object.requestWithoutRetry (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/http-utils.js:92:3)
    at Webdriver._sauceJobUpdate (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:197:13)
    at commands.sauceJobStatus (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/commands.js:2332:24)
    at Webdriver.(anonymous function) [as sauceJobStatus] (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:225:15)
    at EventEmitter.<anonymous> (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:120:29)
    at /Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:116:26
    at cb (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:221:45)
    at /Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:125:17
    at Request._callback (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/http-utils.js:85:7)
    at Request.self.callback (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:364:22)
    at Request.EventEmitter.emit (events.js:98:17)
    at Request.<anonymous> (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:1318:14)
    at Request.init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:367:10)
    at new Request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/request.js:282:8)
    at request (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/node_modules/request/index.js:50:10)
    at Object.requestWithRetry (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/http-utils.js:72:3)
    at Webdriver._init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:103:13)
    at commands.init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/commands.js:33:14)
    at Webdriver.(anonymous function) [as init] (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/node_modules/wd/lib/webdriver.js:225:15)
    at init (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:99:21)
npm ERR! Test failed.  See above for more details.
npm ERR! not ok code 0
rase- commented 9 years ago

Hmm, I haven't seen this in a while. @dagingaa do you think this can be narrowed down to a single browser?

dagingaa commented 9 years ago

My gut feeling is that it isn't browser related directly. In the first log,it seems to happen with Firefox 33, and in the second log it's with Chrome 33 (or so it seems). However, it's hard to tell as the logging is not verbose enough to drill down the exact problem. A suggestion is to add an id to each browser you're testing, and log that with every log message, so you can track which logs belong where.

I don't have time now, but I could try debugging this a little bit more later.

lijunle commented 9 years ago

I got the same error in Travis.

UPDATE I run rebuild in Travis, it run OK then...

lijunle commented 9 years ago

After investigate the log, the exception is thrown by this (bold font) thread, and this (italic font) is the last health thread safe to exit.

... zuul:saucebrowser fetching more results +0ms zuul:saucebrowser waiting for test results from https://vvjityeorm.localtunnel.me/__zuul +226ms zuul:saucebrowser waiting for test results from https://nemmvhkvmb.localtunnel.me/__zuul +756ms _zuul:saucebrowser waiting for test results from https://vjejfjihqv.localtunnel.me/__zuul +19ms_ zuul:saucebrowser res.length: 8 +694ms zuul:saucebrowser done chrome 34 Linux +1ms zuul:saucebrowser finished tests for https://nemmvhkvmb.localtunnel.me/__zuul +1ms zuul:saucebrowser res.length: 8 +41ms zuul:saucebrowser done chrome 33 Mac 10.6 +0ms _zuul:saucebrowser finished tests for https://vjejfjihqv.localtunnel.me/__zuul +1ms_ zuul:saucebrowser quitting browser +1s zuul:saucebrowser quitting browser +34ms zuul:saucebrowser shutdown +494ms - passed: <chrome 33 on Mac 10.6> zuul:saucebrowser running chrome 36 Mac 10.8 +1ms zuul:setup bouncer active on port 60944 +2ms zuul:saucebrowser shutdown +48ms

Error: Not running at Server.close (net.js:1236:11) at Object.shutdown (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/setup.js:78:17) at finish_shutdown (/Users/dagingaa/code/comoyo/appearin-sdk/node_modules/zuul/lib/SauceBrowser.js:206:29) ...

After the last log shutdown from this debug, it wants to shutdown the controller (aka. bouncer), in this line of code. The bouncer is actually a Net server, so the Exception shows the Net server said "Not running", which means bouncer is not running any more.

I have no idea why the bouncer exits before shutdown call. I don't find any clue from the log.

Any ideas?

feross commented 9 years ago

Seeing the same:

Error: Not running
    at Server.close (net.js:1237:11)
    at Object.shutdown (/home/travis/build/feross/webtorrent/node_modules/zuul/lib/setup.js:81:17)
    at finish_shutdown (/home/travis/build/feross/webtorrent/node_modules/zuul/lib/SauceBrowser.js:206:29)
    at /home/travis/build/feross/webtorrent/node_modules/zuul/lib/SauceBrowser.js:235:13
    at cb (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:221:45)
    at null.<anonymous> (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/callbacks.js:23:11)
    at cb (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:160:22)
    at /home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:174:5
    at Request._callback (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/http-utils.js:85:7)
    at Request.self.callback (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/request.js:364:22)
    at Request.init (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/request.js:367:10)
    at new Request (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/request.js:282:8)
    at request (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/index.js:50:10)
    at Object.requestWithRetry (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/http-utils.js:72:3)
    at Webdriver._jsonWireCall (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:171:13)
    at commands.quit (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/commands.js:483:8)
    at Webdriver.(anonymous function) [as quit] (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:225:15)
    at SauceBrowser.shutdown (/home/travis/build/feross/webtorrent/node_modules/zuul/lib/SauceBrowser.js:229:22)
    at Request.init (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/request.js:367:10)
    at new Request (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/request.js:282:8)
    at request (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/index.js:50:10)
    at Object.requestWithoutRetry (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/http-utils.js:92:3)
    at Webdriver._sauceJobUpdate (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:197:13)
    at commands.sauceJobStatus (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/commands.js:2332:24)
    at Webdriver.(anonymous function) [as sauceJobStatus] (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:225:15)
    at EventEmitter.<anonymous> (/home/travis/build/feross/webtorrent/node_modules/zuul/lib/SauceBrowser.js:120:29)
    at /home/travis/build/feross/webtorrent/node_modules/zuul/lib/SauceBrowser.js:116:26
    at cb (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:221:45)
    at /home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:125:17
    at Request._callback (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/http-utils.js:85:7)
    at Request.self.callback (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/request.js:364:22)
    at Request.emit (events.js:98:17)
    at Request.<anonymous> (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/request.js:1318:14)
    at Request.init (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/request.js:367:10)
    at new Request (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/request.js:282:8)
    at request (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/node_modules/request/index.js:50:10)
    at Object.requestWithRetry (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/http-utils.js:72:3)
    at Webdriver._init (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:103:13)
    at commands.init (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/commands.js:33:14)
    at Webdriver.(anonymous function) [as init] (/home/travis/build/feross/webtorrent/node_modules/zuul/node_modules/wd/lib/webdriver.js:225:15)
    at init (/home/travis/build/feross/webtorrent/node_modules/zuul/lib/SauceBrowser.js:99:21)
feross commented 9 years ago

@defunctzombie Might be time to restart localtunnel.me again? ;)

defunctzombie commented 9 years ago

I think it is running.

On Mon, Feb 2, 2015 at 3:12 PM, Feross Aboukhadijeh < notifications@github.com> wrote:

@defunctzombie https://github.com/defunctzombie Might be time to restart localtunnel.me again? ;)

— Reply to this email directly or view it on GitHub https://github.com/defunctzombie/zuul/issues/129#issuecomment-72559213.

feross commented 9 years ago

Hm, retried and it worked this time.

On Mon Feb 02 2015 at 3:15:01 PM Roman Shtylman notifications@github.com wrote:

I think it is running.

On Mon, Feb 2, 2015 at 3:12 PM, Feross Aboukhadijeh < notifications@github.com> wrote:

@defunctzombie https://github.com/defunctzombie Might be time to restart localtunnel.me again? ;)

— Reply to this email directly or view it on GitHub https://github.com/defunctzombie/zuul/issues/129#issuecomment-72559213.

— Reply to this email directly or view it on GitHub https://github.com/defunctzombie/zuul/issues/129#issuecomment-72559536.

parshap commented 9 years ago

I've seen inconsistent results when running tests on Sauce Labs using zuul where once run will result in an error like the above and the next will succeed. This seems to happen especially when testing a large number of browsers. I've had this happen using both localtunnel and Sauce Connect.

I honestly haven't looked into this much, but it sometimes "feels" like it's localtunnel that is dropping the connection and sometimes it "feels" like it's Sauce Labs that is timing out.

lijunle commented 9 years ago

@parshap

I've had this happen using both localtunnel and Sauce Connect.

Do you mean you get this error in the case of only using localtunnel, or in the case of only using Sauce Labs?

If yes, that means zuul should handle this case.

parshap commented 9 years ago

I get different errors depending on if I'm using Sauce Connect or localtunnel.

localtunnel

zuul output stack trace: https://gist.github.com/parshap/8974211384d86622ab4f

Sauce Labs instance that failed as a result: https://saucelabs.com/tests/3dee60b8cf3c4685a44e1fb420d1cdc8

Sauce Connect

zuul output stack trace: https://gist.github.com/parshap/f2c0d514de244340ff38

Sauce Labs instances that failed as a result: https://saucelabs.com/tests/402e850f9a724efeac19eb6306fe3949 https://saucelabs.com/tests/dad812692bbe4186ae35311e30100c76 https://saucelabs.com/tests/7af69358c08641c6ba588162881ab5c7

lijunle commented 9 years ago

@defunctzombie does localtunnel use Sauce Connect under the hood?

defunctzombie commented 9 years ago

@lijunle it does not. Zuul should have nothing to do with sauce connect unless you specifically set it up somehow. I have never used sauce connect.

lijunle commented 9 years ago

@phated I think your Sauce Labs instance link under localtunnel is wrong. It said it is Sauce Connect.

https://saucelabs.com/tests/3dee60b8cf3c4685a44e1fb420d1cdc8

phated commented 9 years ago

@lijunle was that a mistaken @ me?

lijunle commented 9 years ago

@phated sorry, I mentioned a wrong alias... :(

@parshap I think your Sauce Labs instance link under localtunnel is wrong. It said it is Sauce Connect.

https://saucelabs.com/tests/3dee60b8cf3c4685a44e1fb420d1cdc8

parshap commented 9 years ago

@parshap I think your Sauce Labs instance link under localtunnel is wrong. It said it is Sauce Connect.

https://saucelabs.com/tests/3dee60b8cf3c4685a44e1fb420d1cdc8

I think it shows the icon because I had a Sauce Connect connection open at the time of running that test, but if you look at the log you will see it is attempting to go to a localtunnel.me url.

feross commented 9 years ago

Yeah, definitely noticing more errors like this in the past few days... You sure the localtunnel.me server isn't leaking memory and got into a bad state?

On Wed Feb 04 2015 at 1:09:45 AM Parsha Pourkhomami < notifications@github.com> wrote:

@parshap https://github.com/parshap I think your Sauce Labs instance link under localtunnel is wrong. It said it is Sauce Connect.

https://saucelabs.com/tests/3dee60b8cf3c4685a44e1fb420d1cdc8

I think it shows the icon because I had a Sauce Connect connection open at the time of running that test, but if you look at the log you will see it is attempting to go to a localtunnel.me url.

— Reply to this email directly or view it on GitHub https://github.com/defunctzombie/zuul/issues/129#issuecomment-72814706.

defunctzombie commented 9 years ago

localtunnel.me server was crashing a lot. still haven't fixed the issue where every few days it decides to not reboot. That's what I get for trying to use bleeding edge tech like coreos and docker.

vvo commented 9 years ago

getting those errors too, now we need a globally distributed localtunnel :D

vvo commented 9 years ago

So if both localtunnel and sauce connect are failing with a large number of browsers, it's not a localtunnel related bug. More of a saucelabs related bug maybe. And how we handle it.

vvo commented 9 years ago

TL;DR; put higher timeouts..

Well it might be linked to the fact that saucelabs may have started to lower down the ressources of opensauce and that opening browsers will take more time than before.

As for the sauce connect error (https://gist.github.com/parshap/f2c0d514de244340ff38) we should try to put a higher timeout here: https://github.com/vvo/defunctzombie/blob/81a9d1a18481cdcb5d3fb9a2a7db8ec8e8c0f6b1/lib/SauceBrowser.js#L145

And also try to allow more retries here: https://github.com/defunctzombie/zuul/blob/81a9d1a18481cdcb5d3fb9a2a7db8ec8e8c0f6b1/lib/zuul.js#L112

Then for the localtunnel "error" https://gist.github.com/parshap/8974211384d86622ab4f, I am lost, I did not find in our code any "Not running".

But still, why not use the default wd option for configureHttp: https://github.com/admc/wd#http-configuration--base-url more retries, precise timeout.

parshap commented 9 years ago

@vvo thanks for investigating :+1:. I'd be happy to test results with higher timeouts/retries.

defunctzombie commented 9 years ago

I am down for higher timeouts and more retries if it helps.

lijunle commented 9 years ago

@vvo The "Not running" comes from bouncer.close(); (https://github.com/defunctzombie/zuul/blob/master/lib/setup.js#L81)

See my previous comment for details: https://github.com/defunctzombie/zuul/issues/129#issuecomment-72312668

vvo commented 9 years ago

While using sauce_connect (build link) I got a "Not running"

It seems like it was right after finishing a test suite:

2015-03-01-224350_427x94_scrot

This is reproducible, when using sauce_connect, we try to .close() the bouncer after a finished browser and it fails badly. Everytime.

Command line used: zuul --sauce-connect -- test/index.js

@defunctzombie any hint?

vvo commented 9 years ago

Also this was fixed in node 0.12 by https://github.com/joyent/node/pull/7632 but still, why are we trying to close a not listening server? Strange thing is happening here.

vvo commented 9 years ago

Ah, maybe because there's a hidden global variable? https://github.com/defunctzombie/zuul/blob/66547b77f5a975b7daf5cf60c030629f4bb723ca/lib/SauceBrowser.js#L202 :)

vvo commented 9 years ago

How come a such big mistake can happen and make us lose so much time, arggg.

@defunctzombie happy to add a linter? A permissive one, only checking basics like implied globals..

vvo commented 9 years ago

As for the "timeout while opening url" error, it seems to be on @saucelabs side, safari 8 on osx always fails for example right now and @parshap it was also the failing browser for you back in https://github.com/defunctzombie/zuul/issues/129#issuecomment-72814706.

So it may be that safari 8 when using sauceconnect is always a fail, which would be a rather good explanation to our issues.

vvo commented 9 years ago

Linked issue? https://github.com/Automattic/socket.io/issues/1932

vvo commented 9 years ago

End of issue, on some browsers (Safari, IE), wen using sauce connect, you can only access SOME ports on localhost: https://docs.saucelabs.com/reference/sauce-connect/#can-i-access-applications-on-localhost- not all ports. (https://support.saucelabs.com/entries/27401240-Testing-with-a-localhost-server-and-some-browsers-can-t-load-my-website)

It does mean that you better use localtunnel if you want to test all browsers.

Hope my not running fix will be sufficient, I am confident. What we need then is a bigger localtunnel.me maybe at some point, happy to contribute to this @defunctzombie

vvo commented 9 years ago

@rase- this issue can be closed as soon as we release the fix I did

rase- commented 9 years ago

This is hopefully fixed in the latest release.

rase- commented 9 years ago

I'll close this since I haven't seen or heard of this happening lately. Please do reopen if the errors come back, though.