Intern unit tests intermittently hang against BrowserStack/IE

[wkeese]

When using Intern to run a unit test against IE in BrowserStack, the unit test seems to complete but the intern command never returns. Instead it keeps resending the following URL once a minute:

Get URL⇒ http://localhost:9000/__intern/client.html?config=deliteful%2Ftests%2Fintern.browserstack-ie&reporters=webdriver&baseUrl=%2F&sessionId=133633c91ec11dbbb9fa9904476866c2dab4a9a4

Happens intermittently w/the latest intern (2.0.3).

To reproduce:

$ git clone https://github.com/wkeese/deliteful.git
$ cd deliteful
$ git checkout intern2
$ npm install
$ bower install
$ export BROWSERSTACK_USERNAME=...
$ export BROWSERSTACK_ACCESS_KEY=...
$ grunt test:browserstack-ie

(except filling in your user name and access key, as shown on https://www.browserstack.com/automate)

You can check https://www.browserstack.com/automate to see the progress. It will intermittently hang, so if the final command completes for you, trying running it a few more times until you see the problem.

[csnover]

URL retrieval requests are the heartbeat command to prevent the connection from being closed during unit tests, which do not communicate via the WebDriver channel. A failure to transmit a /client/end message from the remote unit test suite will cause the runner to never proceed to functional testing.

One possible mitigation strategy to avoid an infinite wait would be to use an execute call instead of a URL retrieval to poll for a property of the unit testing system, to verify whether or not it thinks it is still running. This will result in the maximum duration of a failed unit test run to be equal to the length of the heartbeat interval, assuming that the unit test framework has not failed so catastrophically that it is not capable of even updating a flag. However, it will not fix the intermittent test failure issue; if the heartbeat interval becomes responsible for killing the unit testing session, that would need to be recorded as a failure, since it is outside the normal operating parameters of the system.

[csnover]

(Also, any intermittent failure with a service provider like this is likely to be an issue with the provider and not anything that can be addressed with Intern, but if you can provide any evidence or detail that this is not the case that would be great.)

[Blasz]

I have also experienced this when I did not have the intern proxy url configured correctly. Trying to run the unit tests resulted in 404's and the 'get current url' heartbeat resulted in an infinite wait.

[wkeese]

One possible mitigation strategy to avoid an infinite wait would be to use an execute call instead of a URL retrieval to poll for a property of the unit testing system, to verify whether or not it thinks it is still running.

I guess what I expected was simply that the client would call webdriver's executeAsync() to run the unit test on the server (i.e. in the browser), and executeAsync() would presumably internally poll a few times a second until the test was finished.

With the current code, I can confirm through logging that intern/lib/reporters/webdriver.js is calling publish("/client/end") yet the client is not receiving it. (I see the logs by pressing the "interactive session" link on BrowserStack and then doing F12.) Pluswhich, manually publishing the topic from developer tools doesn't work either (i.e. require(["dojo/topic"], function(topic){ topic.publish("/client/end", 12345); })) I don't understand how you pass messages from the browser back to the client so I can't debug any further.

[jason0x43]

Messages are POSTed from the client back to the test proxy, which in turn publishes them for the test runner to consume. What I've seen happen in the past is that at some point BrowserStack's localhost proxy returns a 404 for one of the client messages (something before "/client/end"). The test proxy will only publish messages to the runner when it has a continuous series of messages. If it receives sequence numbers 1, 2, 3, 5, 6, 7, 8, and 9 (assume 9 is "/client/end"), it won't publish 5-9 until 4 has been received. The proxy (and the test runner) will wait indefinitely for message 4 to be received.

It seemed like BrowserStack had taken care of this problem the last time I tested it. Maybe not.

[csnover]

@jason0x43 If you find some time in the next couple weeks could you reach out to @BrowserStack about this? (Or maybe they notice the @mention and come around :)) I am not sure what we can do about this… 404 is technically not a permanent error, but I don’t feel like hacking in retransmission to the reporter since the provider should be guaranteeing correct transit through their tunnel.

[jason0x43]

Sure, I'll look into it.

I'm not sure what the downside of having the client retransmit would be, though. I mean, it shouldn't be necessary...but it would make the testing process more robust without breaking anything else.

[csnover]

The downside is that I really dislike having to write more code because other people can’t write code that works. :((((((((((

No, any downside is mostly just an argument based around the accrual of technical debt and the best use of limited resources.

1. It is easier to solve a problem closer to the source. Stop dumping waste into the river, don’t try to filter the sewage 100 miles downstream
2. It takes a lot of time to figure out what is wrong and reliably fix it, which takes away from what little valuable time we have that we could spend on amazing new features and dealing with our own technical debt
3. In some cases (probably not this one), we accidentally break things when it is fixed upstream due to an edge condition in the workaround that we failed to account for (shout out to my homey user-agent sniffing)

There are some secondary arguments about slippery slopes (what is the line that we absolutely do not cross, and who gets to make that judgment?) and giving unfair preferential treatment to groups just because they are large, rather than because they are good (a personal pet peeve) but they don’t apply directly to this situation.

[csnover]

I haven’t heard about this being a problem in quite a while so I am going to close this. Please feel free to open a new ticket or reply with reproduction case if this is still a problem!