Magellan not returning proper exit code for failed runs with saucelabs executor v4.10.0

Hi there,

We've been using magellan to parallelize our nightwatch runs on Saucelabs. After bumping the version of saucelabs executor to 4.10.0 from 1.0.5, we noticed that failing runs were no longer exiting with a status code of 1 to indicate a failure. This meant that failed test runs were actually reporting as successful in our buildkite pipelines, which rely on this exit code being accurate.

Here is example output running a simple magellan command with a test that is designed to fail. I should also note that we are not actually using the saucelabs executor to define the browser but are rather using a locally-defined browser from our nightwatch.json file.

Here is the browser definition in nightwatch.json:

"saucelabs_win10_firefox47" : {
      "selenium_port" : 80,
      "selenium_host" : "ondemand.saucelabs.com",
      "desiredCapabilities": {
        "browserName": "firefox",
        "version": "47.0",
        "platform": "Windows 10",
        "javascriptEnabled": true,
        "screenResolution": "1920x1200",
        "elementScrollBehavior": 1
      }
    },

And this is the command I'm using to run the test:

magellan --test=tests/sample/fail.js --local_browsers=saucelabs_win10_firefox47

With Saucelabs executor at 1.0.5

[INFO] [Magellan]
[INFO] [Magellan] ============= Suite Complete =============
[INFO] [Magellan]      Status: FAILED
[INFO] [Magellan]     Runtime: 42.9s
[INFO] [Magellan] Total tests: 1
[INFO] [Magellan]  Successful: 0 / 1
[INFO] [Magellan]      Failed: 1 / 1
error Command failed with exit code 1.
jtheodore@MacBook-Pro-2 (jt/nw-fix-false-negative-runs *) ~/source/ag.js: echo $?
1

With Saucelabs executor at 4.10.0

[INFO] [Magellan]
[INFO] [Magellan] ============= Suite Complete =============
[INFO] [Magellan]      Status: FAILED
[INFO] [Magellan]     Runtime: 40s
[INFO] [Magellan] Total tests: 1
[INFO] [Magellan]  Successful: 0 / 1
[INFO] [Magellan]      Failed: 1 / 1
jtheodore@MacBook-Pro-2 (jt/nw-fix-false-negative-runs *) ~/source/ag.js: echo $?
0

TestArmada / magellan-saucelabs-executor

Magellan not returning proper exit code for failed runs with saucelabs executor v4.10.0 #18