SauceLabsTunnel not closing once tests are done

[rodneyrehm]

Hey,

after executing all the tests the runner hangs without any new output to the console. SauceLabs says there is a tunnel open, but no VMs are running anymore. When I kill the runner, I get the following printed to console:

^C
Shutting down gracefully; please wait or hit CTRL+C again to quit immediately

0/0 tests failed
192/192 tests failed
SUITE ERROR
Error: Test did not see a new command for 60 seconds. Timing out.
  at <node_modules/intern/node_modules/digdug/SauceLabsTunnel.js:339:12>
  at <node_modules/intern/node_modules/dojo/Promise.ts:393:15>
  at runCallbacks  <node_modules/intern/node_modules/dojo/Promise.ts:11:11>
  at <node_modules/intern/node_modules/dojo/Promise.ts:317:4>
  at run  <node_modules/intern/node_modules/dojo/Promise.ts:237:7>
  at <node_modules/intern/node_modules/dojo/nextTick.ts:44:3>
  at process._tickCallback  <node.js:355:11>
  at Command.then  <node_modules/intern/node_modules/leadfoot/Command.js:542:10>
  at Command  <node_modules/intern/node_modules/leadfoot/Command.js:566:15>
  at Suite.teardown  <node_modules/intern/lib/executors/Runner.js:209:36>
  at <node_modules/intern/lib/Suite.js:363:36>
  at new Promise  <node_modules/intern/node_modules/dojo/Promise.ts:411:3>
  at teardown  <node_modules/intern/lib/Suite.js:362:12>
  at <node_modules/intern/node_modules/dojo/Promise.ts:393:15>
  at runCallbacks  <node_modules/intern/node_modules/dojo/Promise.ts:11:11>
  at <node_modules/intern/node_modules/dojo/Promise.ts:317:4>
  at run  <node_modules/intern/node_modules/dojo/Promise.ts:237:7>

I'm not sure if this is a problem with DigDug or Intern, so I'm throwing it in here…

[csnover]

Hm. It looks like one of the VMs shut down in the middle of testing because the server thought it was not receiving any Selenium commands. Then the service sends that error when the final state of the session is set. I don’t know why that would happen unless you have a test that never resolves. A reproducible test case or more information about when this happens (all the time? only when you have certain tests enabled?) would be useful.

[rodneyrehm]

For OSS all tests are public. I seem to be hitting two issues:

• Test did not see a new command for 60 seconds. Timing out.
• Test exceeded maximum duration after 1800 seconds

In both cases all tests pass when opening node_modules/intern/client.html?config=test/browser in the respective browser. The same is true for the tests IE10 and Firefox dev - they're all green when run via the client.html, but fail in the Intern/SauceLabs stack. I have no idea where to begin debugging this.

[rodneyrehm]

I was still running 3.0.0-rc.3, let me update to 3.0.0 and try again.

[rodneyrehm]

Same issue with 3.0.0 - will wait for #466 to land as suggested in IRC.

[csnover]

@rodneyrehm 3.0.1 is released with fixes for #466, can you verify this is no longer broken in the new release? Thanks!

[rodneyrehm]

The process exits on its own. But I'm getting the following errors and no success/failure output, only errors along the lines of:

SUITE ERROR
Error: Cannot find the Node.js require
@http://localhost:9000/__intern/node_modules/dojo/node.js:14:1
  at execModule  <__intern/node_modules/dojo/loader.js:332:54>
  at checkComplete/<  <__intern/node_modules/dojo/loader.js:374:21>
  at guardCheckComplete  <__intern/node_modules/dojo/loader.js:361:9>
  at checkComplete  <__intern/node_modules/dojo/loader.js:366:1>
  at injectModule/onLoadCallback  <__intern/node_modules/dojo/loader.js:428:17>
  at injectUrl/handler  <__intern/node_modules/dojo/loader.js:514:65>
0/0 tests failed

SUITE ERROR
Error: Error: Cannot find the Node.js require at http://localhost:9000/__intern/node_modules/dojo/node.js:14:59
  at <__intern/lib/executors/PreExecutor.js:294:32>
  at dispatcher  <__intern/node_modules/dojo/aspect.js:66:47>
0/0 tests failed

Error: Cannot find the Node.js require
  at <__intern/node_modules/dojo/node.js:14:15>
  at execModule  <__intern/node_modules/dojo/loader.js:332:62>
  at <__intern/node_modules/dojo/loader.js:374:21>
  at guardCheckComplete  <__intern/node_modules/dojo/loader.js:361:9>
  at checkComplete  <__intern/node_modules/dojo/loader.js:366:32>
  at onLoadCallback  <__intern/node_modules/dojo/loader.js:428:17>
  at HTMLScriptElement.handler  <__intern/node_modules/dojo/loader.js:514:65>

SUITE ERROR
UnknownError: [POST http://(redacted)@localhost:4444/wd/hub/session / {"desiredCapabilities":{"browserName":"firefox","version":"dev","platform":"Linux","tunnel-identifier":"1441177503890","name":"test/sauce","idle-timeout":60}}] The Sauce VMs failed to start the browser or device
For more info, please check https://docs.saucelabs.com/reference/troubleshooting-common-error-messages
  at Server.createSession  <node_modules/intern/node_modules/leadfoot/Server.js:307:15>
  at <node_modules/intern/lib/executors/Runner.js:189:22>
  at retry  <node_modules/intern/lib/util.js:717:13>
  at <node_modules/intern/node_modules/dojo/Promise.ts:393:15>
  at runCallbacks  <node_modules/intern/node_modules/dojo/Promise.ts:11:11>
  at <node_modules/intern/node_modules/dojo/Promise.ts:317:4>
  at run  <node_modules/intern/node_modules/dojo/Promise.ts:237:7>
  at <node_modules/intern/node_modules/dojo/nextTick.ts:44:3>
  at process._tickCallback  <node.js:355:11>
0/0 tests failed

Error: Run failed due to one or more suite errors
  at emitLocalCoverage  <node_modules/intern/lib/executors/Executor.js:342:18>
  at finishSuite  <node_modules/intern/lib/executors/Executor.js:359:15>
  at <node_modules/intern/lib/executors/Executor.js:366:7>
  at <node_modules/intern/node_modules/dojo/Promise.ts:393:15>
  at runCallbacks  <node_modules/intern/node_modules/dojo/Promise.ts:11:11>
  at <node_modules/intern/node_modules/dojo/Promise.ts:317:4>
  at run  <node_modules/intern/node_modules/dojo/Promise.ts:237:7>
  at <node_modules/intern/node_modules/dojo/nextTick.ts:44:3>
  at process._tickCallback  <node.js:355:11>

I'm not sure what's going on with dojo and the missing console reporter - as far as I can tell it's configured properly in sauce.js and intern.base-config.js. The dojo thing is particularly weird considering there are successful tests on sauce.

[csnover]

I don’t see anything specifically about Dojo (outside of it’s the library that Intern uses) or the Console reporter in there; I see that your module IDs are incorrect in your various configuration files (modules IDs do not end with “.js”), so the first thing to do would be to fix those, and then the second thing would be to breakpoint inside of dojo/node to figure out what is trying to use it in the browser.

[csnover]

I got a similar report from a second person, I think that reporter information might be leaking into the client that is supposed to be overwritten entirely instead of merged.

[rodneyrehm]

I don’t see anything specifically about Dojo

The fix you provided to #491 solved the node require issue for me too. Waiting for 3.0.3 to properly retest

[rodneyrehm]

I've retested with 3.0.3 and it now seems to be running for the most part. The process still hangs at the end and the remaining VMs are not closed.

I'm getting the following error (repeatedly):

SUITE ERROR
UnknownError: [POST http://(redacted)@localhost:4444/wd/hub/session/2ee6a2bad2754287a342909b4b2bbf99/url / {"url":"http://localhost:9000/__intern/client.html?config=test%2Fsauce&basePath=%2F&initialBaseUrl=%2F&reporters=%7B%22id%22%3A%22WebDriver%22%7D&rootSuiteName=firefox%20dev%20on%20OS%20X%2010.10%20-%20unit%20tests&sessionId=2ee6a2bad2754287a342909b4b2bbf99"}] Due to a previous error, this job has already finished. You can learn more at https://saucelabs.com/jobs/2ee6a2bad2754287a342909b4b2bbf99
  at ProxiedSession._post  <node_modules/intern/node_modules/leadfoot/Session.js:59:30>
  at ProxiedSession.Session.get  <node_modules/intern/node_modules/leadfoot/Session.js:523:15>
  at <node_modules/intern/lib/ProxiedSession.js:86:25>
  at getFinalValue  <node_modules/intern/node_modules/dojo/Promise.ts:462:21>
  at <node_modules/intern/node_modules/dojo/Promise.ts:482:10>
  at <node_modules/intern/node_modules/dojo/Promise.ts:393:15>
  at runCallbacks  <node_modules/intern/node_modules/dojo/Promise.ts:11:11>
  at <node_modules/intern/node_modules/dojo/Promise.ts:317:4>
  at run  <node_modules/intern/node_modules/dojo/Promise.ts:237:7>
  at <node_modules/intern/node_modules/dojo/nextTick.ts:44:3>
  at Command.target.(anonymous function) [as get]  <node_modules/intern/node_modules/leadfoot/Command.js:610:11>
  at ClientSuite.lang.mixin.run  <node_modules/intern/lib/ClientSuite.js:149:6>
  at runWithCatch  <node_modules/intern/lib/Suite.js:314:21>
  at next  <node_modules/intern/lib/Suite.js:322:18>
  at <node_modules/intern/lib/Suite.js:345:6>
  at new Promise  <node_modules/intern/node_modules/dojo/Promise.ts:411:3>
  at runTests  <node_modules/intern/lib/Suite.js:278:12>
  at <node_modules/intern/node_modules/dojo/Promise.ts:393:15>
  at runCallbacks  <node_modules/intern/node_modules/dojo/Promise.ts:11:11>
  at <node_modules/intern/node_modules/dojo/Promise.ts:317:4>
0/0 tests failed

may be connected to bad selenium version (according to SauceLabs docs) - not that I'm specifying any version at all (sauce config, base config).

---

And this error exactly once (in one of the Safaris):

SUITE ERROR
Error: TypeError: undefined is not a function (evaluating 'obj.GetPluginStatus()') at undefined:1:88
  at <__intern/lib/executors/PreExecutor.js:306:32>
  at dispatcher  <__intern/node_modules/dojo/aspect.js:66:47>
268/268 tests failed

[csnover]

I am sure Firefox Developer Edition isn’t supported by Selenium project; Mozilla changes APIs in almost every new Firefox release, and these changes break Selenium, so you’re on your own if you want to try to target that environment (at least until Mozilla start maintaining their own stable WebDriver server). I’m not saying that sessions should hang in Intern, ever, but for your own edification you’re never going to get stable testing with developer edition of Firefox so you should probably stop trying.

[rodneyrehm]

Thanks for the hint! I removed Firefox Dev and reduced the problem to the second error (the Safari one) - the process still hangs until I ^C it.