Passing unsupported OS/Browser combinations through to saucelabs causes process to hang and never handle the result

[tejacques]

I dug into the details here, and basically here's what's going on:

The default number of fetch attempts to get the status is effectively infinity. Now, when there's a browser config issue, the test is never started, and the result that it gets back is not handled correctly:

$ grunt test:broken
Adding environment variables
Running "test:broken" (test) task

Running "connect:server" (connect) task
Started connect web server on http://0.0.0.0:9000

Running "saucelabs-mocha:broken" (saucelabs-mocha) task
=> Starting Tunnel to Sauce Labs
>> Connected to Saucelabs

 1 / 1 tests started
Result of test:  { status: 'test queued',
  platform: [ 'OSX 10.6', 'safari', '5.1' ],
  id: '4c73b921ab4c40aeab7d8fb19e422d20',
  job_id: 'job not ready' }
Result of test:  { status: 'test error',
  platform: [ 'OSX 10.6', 'safari', '5.1' ],
  id: '4c73b921ab4c40aeab7d8fb19e422d20',
  job_id: 'job not ready' }

Even though the status is 'test error', status is never read. It relies purely on job_id and continues running until it gets back an alphanumeric value.

The fix here is pretty simple. result.status just needs to be checked to see if it is 'test error'

src/Job.js:156

if (result.status !== 'test error' && (!body.completed || !reJobId.test(jobId))) {
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                       ^

A little more might need to be done to get the reporting to line up though, but that at least allows it to pass.

[tejacques]

There's one other change I made, which is to not throw an error when result.status === 'test error'

I did this by removing this if block: src/Job.js:111

        if (result.status === 'test error') {
          // A detailed error message should be composed here after the Sauce Labs API is
          // modified to report errors better, see #102.
          throw new Error('Test Error');
        }

I think the state of what this returns has improved since that bug was filed.

Without this commented out I get this:

$ grunt test:broken
Adding environment variables
Running "test:broken" (test) task

Running "connect:server" (connect) task
Started connect web server on http://0.0.0.0:9000

Running "saucelabs-mocha:broken" (saucelabs-mocha) task
=> Starting Tunnel to Sauce Labs
>> Connected to Saucelabs

 1 / 1 tests started

Tested http://127.0.0.1:9000/test/mocha_test.html
Platform: undefined
Passed: false
Url undefined
>> All tests completed with status false
=> Stopping Tunnel to Sauce Labs
Warning: Task "saucelabs-mocha:broken" failed. Use --force to continue.

Aborted due to warnings.

With the block commented out I get this:

$ grunt test:broken
Adding environment variables
Running "test:broken" (test) task

Running "connect:server" (connect) task
Started connect web server on http://0.0.0.0:9000

Running "saucelabs-mocha:broken" (saucelabs-mocha) task
=> Starting Tunnel to Sauce Labs
>> Connected to Saucelabs

 1 / 1 tests started

Tested http://127.0.0.1:9000/test/mocha_test.html
Platform: OSX 10.6,safari,5.1
Passed: false
Url undefined
>> All tests completed with status false
=> Stopping Tunnel to Sauce Labs
Warning: Task "saucelabs-mocha:broken" failed. Use --force to continue.

Aborted due to warnings.

[kevin-brown]

I've been running into similar issues with Select2 that I just tracked down to Firefox builds not actually being started.

So we're having a similar issue (process is hanging because of a (possibly) bad configuration) but the test isn't being triggered at all so the result isn't going to come back correctly.

Edit: The issue was that I wasn't locking the platform down to anything for Firefox builds, locking it to "linux" fixed the issue.

[alexan]

thanks a lot kevin this fixed my issue with firefox as well. But the project should throw a better error description if possible.

[Jonahss]

Sure, anyone want to submit a Pull Request?