Karma times out on Travis/SL because IE9 is never started (Angular.js build)

[juliemr]

The angular.js unit test job fails intermittently because Karma times out (hits the Travis 10 minute timeout). It appears that IE9 is never started, because it simply doesn't appear in the logs.

Passing log (https://travis-ci.org/angular/angular.js/jobs/46478813)

Running "tests:jqlite" (tests) task
INFO [karma]: Karma v0.12.23 server started at http://localhost:9876/
INFO [launcher]: Starting browser chrome 34 on SauceLabs
INFO [launcher]: Starting browser safari 7 (OS X 10.9) on SauceLabs
INFO [launcher]: Starting browser firefox 26 on SauceLabs
INFO [launcher]: Starting browser internet explorer 9 (Windows 2008) on SauceLabs
INFO [launcher]: Starting browser internet explorer 10 (Windows 2012) on SauceLabs
INFO [launcher]: Starting browser internet explorer 11 (Windows 8.1) on SauceLabs
INFO [launcher.sauce]: safari 7 (OS X 10.9) session at https://saucelabs.com/tests/a1c585d795724a029bae36e67e31e35a
INFO [launcher.sauce]: internet explorer 10 (Windows 2012) session at https://saucelabs.com/tests/48f533b9bc1949eda49e45262229cded
INFO [launcher.sauce]: firefox 26 session at https://saucelabs.com/tests/ee087fbd985e42648ed2872152407e42
INFO [launcher.sauce]: chrome 34 session at https://saucelabs.com/tests/f3051034eeaf4a4887afa5fe6fd6e170
INFO [launcher.sauce]: internet explorer 9 (Windows 2008) session at https://saucelabs.com/tests/371e0addb30745e68b8d07d20da18819
INFO [launcher.sauce]: internet explorer 11 (Windows 8.1) session at https://saucelabs.com/tests/8f1b06bc45ae44adaf3c3c2fb63d146a
INFO [Safari 7.0.6 (Mac OS X 10.9.5)]: Connected on socket BAZnH81w0QVEU9d6BN6V with id 25857773
INFO [IE 10.0.0 (Windows 8)]: Connected on socket PZYKbkisMEDv3DboBN6W with id 68715838
INFO [Chrome 34.0.1847 (Linux)]: Connected on socket oSRwCTvd4ozO4EpRBN6X with id 28979983
INFO [Firefox 26.0.0 (Linux)]: Connected on socket A-MQY3emFdys7dgBBN6Y with id 73423148
INFO [IE 9.0.0 (Windows 7)]: Connected on socket bUia3MuchXVb8FM2BN6Z with id 87881399
INFO [IE 11.0.0 (Windows)]: Connected on socket FHeqpUIHfa2odFh6BN6a with id 81051024

Failing log (https://travis-ci.org/angular/angular.js/jobs/46487145)

Running "tests:jqlite" (tests) task
INFO [karma]: Karma v0.12.23 server started at http://localhost:9876/
INFO [launcher]: Starting browser chrome 34 on SauceLabs
INFO [launcher]: Starting browser safari 7 (OS X 10.9) on SauceLabs
INFO [launcher]: Starting browser firefox 26 on SauceLabs
INFO [launcher]: Starting browser internet explorer 9 (Windows 2008) on SauceLabs
INFO [launcher]: Starting browser internet explorer 10 (Windows 2012) on SauceLabs
INFO [launcher]: Starting browser internet explorer 11 (Windows 8.1) on SauceLabs
INFO [launcher.sauce]: firefox 26 session at https://saucelabs.com/tests/65b1306e9603437d807d9805d209911e
INFO [launcher.sauce]: chrome 34 session at https://saucelabs.com/tests/ef5ec30488de4f7c9b303eec001a68ae
INFO [launcher.sauce]: safari 7 (OS X 10.9) session at https://saucelabs.com/tests/695162d0c12f496f85480177014190ac
INFO [launcher.sauce]: internet explorer 10 (Windows 2012) session at https://saucelabs.com/tests/aa12eb2e46364e859a48f40851d4be8d
INFO [launcher.sauce]: internet explorer 9 (Windows 2008) session at https://saucelabs.com/tests/b98c944eb4104602bd964640e546c04e
INFO [launcher.sauce]: internet explorer 11 (Windows 8.1) session at https://saucelabs.com/tests/56bb42dcd1164cf1a110228e2f5f48ca
INFO [Chrome 34.0.1847 (Linux)]: Connected on socket qKTu-N5jvosEdGSVRkRn with id 36441444
INFO [Firefox 26.0.0 (Linux)]: Connected on socket SOhY9UooQP5zBonvRkRo with id 28659724
INFO [Safari 7.0.6 (Mac OS X 10.9.5)]: Connected on socket f_CEKdzLJ4XskgpIRkRp with id 91457420
INFO [IE 10.0.0 (Windows 8)]: Connected on socket 1vcnyNDBu7vGh5EJRkRq with id 56690742

Note that the failing log never connects to IE9, but it doesn't show an error message either. The Sauce Labs link for the IE9 session is https://saucelabs.com/tests/b98c944eb4104602bd964640e546c04e

[plasticine]

We’re seeing exactly this at work with a large Karma test suite and Saucelabs. I’ve not yet been able to work out if the issue is with Karma, Saucelabs or karma-saucelabs-launcher.

[DylanLacey]

I'm a Sauce Labs employee, investigating the same issue for a customer.

Something weird seems to be going on; When I watched one of the tests with video turned on (By adding the line recordVideo: true to the sauceLabs object in my config) the Karma header never updated from 'Karma - starting', but the list of browsers and browser activity did.

From https://github.com/karma-runner/karma/blob/v0.12.31/client/updater.js#L22 it appears that, once a connection has been made, the status should have been updated.

But, the list of browsers is updated when an 'info' event is emitted: https://github.com/karma-runner/karma/blob/v0.12.31/client/updater.js#L27

(Both of these were in karma.js which I assume is compiled from everything in the client directory?)

So I'm a little confused how we could simultaneously not be connected, and receiving events from the server. I'd think it was just the StatusUpdater being naughty but you were saying you're not seeing a console log of the socket ever connecting.

In a local install of Karma I added the connecting and connect_failed events to try to track more precisely what was going on. Unfortunately I can't get the problem to reproduce. I've cloned Angular's repo and am currently trying to get it running tests in the hope it'll fail consistently, so I can have at it more savagely.

Disclaimer: I'm mostly a Ruby dude, I don't have the JS-fu so might just be missing something obvious.

[mnichols]

Also trying Win7 + IE 9 and I'm seeing (in screenshot) a 'Bad Gateway: Connection refused' inside the browser when the POST to /session happens . I tried this config earlier with a different test runner and had similar connection problems with sauce (using saucie + testem). That seems to indicate a sauce issue.

[dignifiedquire]

@juliemr is this still an issue?

[wesleycho]

Looking at the code path and seeing what is said from prior attempts to debug in comments here, my suspicion would be that a V8 bug is lurking here (although I do not know if it would be fixed already), or Socket.io isn't doing the right thing in IE9 here.

On register, socket.io creates a Browser instance here, which then adds it to the collection when init is called here. add just pushes it into the internal array and emits the change event here. Lastly, the change listener simply emits the event to the client with the data here.

The only way I can see that a connection could fail here is if the array is not truly mutated before the serialize method is called, which would explain why that socket is not notified of the info event.