search

nightwatchjs / nightwatch

Integrated end-to-end testing framework written in Node.js and using W3C Webdriver API. Developed at @browserstack
https://nightwatchjs.org
MIT License
11.83k stars 1.32k forks source link

Mocha runner crashes when a test throws an exception (leading to BrowserStack timeouts) #2371

Closed JRoseDev closed 4 years ago

JRoseDev commented 4 years ago

Describe the bug

When using the mocha runner, tests that throw cause the whole test process to crash, leaving timeouts in BrowserStack with no indication of the reason. This causes us a lot of issues because we can't easily find out why tests failed, and it increases the amount of time taken for the run to complete.

This is the same issue as https://github.com/nightwatchjs/nightwatch/issues/2081.

detail

The problem is that [lib\runner\test-runners\mocha.js:95](https://github.com/nightwatchjs/nightwatch/blob/master/lib/runner/test-runners/mocha.js#L95) chains off the return of `createRunnable`: ```js this.parent.nightwatchSuite.createRunnable(this.title, () => { this.fn.apply(ctx, args); }).then(err => { queueFinished = true; if (!isHook) { done(); } }).catch(err => { queueFinished = true; this.parent.nightwatchSuite.terminate('FAILED') .then(_ => done(err)).catch(err => done(err)); }); ``` But [/lib/testsuite/runnable.js:122](https://github.com/nightwatchjs/nightwatch/blob/master/lib/testsuite/runnable.js#L122) (in the case of mocha + BrowserStack at least) can return the result of `rejectFn`: ```js try { result = this.runFn(); } catch (err) { return this.deffered.rejectFn(err); } ``` Which always returns `undefined` [/lib/testsuite/runnable.js:76](https://github.com/nightwatchjs/nightwatch/blob/master/lib/testsuite/runnable.js#L76): ```js this.deffered.rejectFn = (err, force = false) => { if (!this.deffered.settled || force) { reject(err); } this.deffered.settled = true; }; ``` So the initial error is caught, but `mocha.js` throws: ``` TypeError: Cannot read property 'then' of undefined ``` which crashes the whole test process.

In my testing returning the rejected promise from rejectFn prevents the unhandled exception, and the error from the test is correctly handled by the catch from the first snippet: https://github.com/JRoseDev/nightwatch/commit/7447c2db7ee7f92b99bec0822aff073b52e987ec

I'm happy to raise a PR if you're OK with that?

Sample test

sampleTest.js

```js describe("Catching unhandled exceptions from tests", function() { it("a test that fails", function(browser) { throw new Error("This should be recorded as a test failure"); }); }); ```

Run with command

$ nightwatch test/sampleTest.js -e browserstack.chrome

Verbose output

debug.log

```txt - Connecting to hub-cloud.browserstack.com on port 443... Request POST https://hub-cloud.browserstack.com /wd/hub/session { desiredCapabilities: { browserName: 'chrome', platform: 'ANY', chromeOptions: { w3c: false }, 'bstack:options': { local: 'false', userName: '', accessKey: '' }, 'browserstack.key': '', 'browserstack.user': '', / Connecting to hub-cloud.browserstack.com on port 443... Response 200 POST https://hub-cloud.browserstack.com/wd/hub/session (3115ms) { state: null, sessionId: 'b3589494f69f2aae5fe6909f60d8ccd25c840767', value: { mobileEmulationEnabled: false, timeouts: { implicit: 0, pageLoad: 300000, script: 30000 }, hasTouchScreen: false, platform: 'WINDOWS', acceptSslCerts: false, 'goog:chromeOptions': { debuggerAddress: 'localhost:1491' }, acceptInsecureCerts: false, webStorageEnabled: true, browserName: 'chrome', takesScreenshot: true, javascriptEnabled: true, setWindowRect: true, unexpectedAlertBehaviour: 'ignore', applicationCacheEnabled: false, rotatable: false, networkConnectionEnabled: false, chrome: { chromedriverVersion: '80.0.3987.16 (320f6526c1632ad4f205ebce69b99a062ed78647-refs/branch-heads/3987@{#185})', userDataDir: 'C:\\Windows\\proxy\\scoped_dir6624_1163261824' }, takesHeapSnapshot: true, pageLoadStrategy: 'normal', strictFileInteractability: false, databaseEnabled: false, handlesAlerts: true, version: '80.0.3987.132', browserConnectionEnabled: false, nativeEvents: true, 'webdriver.remote.sessionid': 'b3589494f69f2aae5fe6909f60d8ccd25c840767', locationContextEnabled: true, cssSelectorsEnabled: true }, i Connected to hub-cloud.browserstack.com on port 443 (3199ms). Using: chrome (80.0.3987.132) on WINDOWS platform. Received session with ID: b3589494f69f2aae5fe6909f60d8ccd25c840767 Catching unhandled exceptions from tests unhandledRejection: Error: This should be recorded as a test failure at Context. (C:\Users\Joshua.Rose\source\repos\nightwatch-repro\tests.js:3:15) at Runnable.parent.nightwatchSuite.createRunnable [as __runFn] (C:\Users\Joshua.Rose\source\repos\nightwatch-repro\node_modules\nightwatch\lib\runner\test-runners\mocha.js:98:21) at Runnable.run (C:\Users\Joshua.Rose\source\repos\nightwatch-repro\node_modules\nightwatch\lib\testsuite\runnable.js:123:21) at TestSuite.createRunnable (C:\Users\Joshua.Rose\source\repos\nightwatch-repro\node_modules\nightwatch\lib\testsuite\index.js:441:33) at parent.nightwatchSuite.createSession.then._ (C:\Users\Joshua.Rose\source\repos\nightwatch-repro\node_modules\nightwatch\lib\runner\test-runners\mocha.js:97:39) Error: This should be recorded as a test failure at Context. (C:\Users\Joshua.Rose\source\repos\nightwatch-repro\tests.js:3:15) unhandledRejection: TypeError: Cannot read property 'then' of undefined at parent.nightwatchSuite.createSession.then._ (C:\Users\Joshua.Rose\source\repos\nightwatch-repro\node_modules\nightwatch\lib\runner\test-runners\mocha.js:99:13) TypeError: Cannot read property 'then' of undefined ```

Configuration

nightwatch.json

```js // Autogenerated by Nightwatch // Refer to the online docs for more details: https://nightwatchjs.org/gettingstarted/configuration/ const Services = {}; loadServices(); module.exports = { // An array of folders (excluding subfolders) where your tests are located; // if this is not specified, the test source must be passed as the second argument to the test runner. src_folders: [], // See https://nightwatchjs.org/guide/working-with-page-objects/ page_objects_path: "", // See https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-commands custom_commands_path: "", // See https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-assertions custom_assertions_path: "", // See https://nightwatchjs.org/guide/#external-globals globals_path: "", end_session_on_fail: true, webdriver: {}, test_settings: { default: { disable_error_log: false, launch_url: "https://nightwatchjs.org", screenshots: { enabled: false, path: "screens", on_failure: true }, desiredCapabilities: { browserName: "firefox" }, webdriver: { start_process: true, server_path: Services.geckodriver ? Services.geckodriver.path : "" } }, firefox: { desiredCapabilities: { browserName: "firefox", alwaysMatch: { // Enable this if you encounter unexpected SSL certificate errors in Firefox // acceptInsecureCerts: true, "moz:firefoxOptions": { args: [ // '-headless', // '-verbose' ] } } }, webdriver: { start_process: true, port: 4444, server_path: Services.geckodriver ? Services.geckodriver.path : "", cli_args: [ // very verbose geckodriver logs // '-vv' ] } }, chrome: { desiredCapabilities: { browserName: "chrome", chromeOptions: { // This tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78) // w3c: false, // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/ args: [ //'--no-sandbox', //'--ignore-certificate-errors', //'--allow-insecure-localhost', //'--headless' ] } }, webdriver: { start_process: true, port: 9515, server_path: Services.chromedriver ? Services.chromedriver.path : "", cli_args: [ // --verbose ] } }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using the browserstack.com cloud service | // | // Please set the username and access key by setting the environment variables: | // - BROWSERSTACK_USER | // - BROWSERSTACK_KEY | // .env files are supported | ////////////////////////////////////////////////////////////////////////////////// browserstack: { selenium: { host: "hub-cloud.browserstack.com", port: 443 }, // More info on configuring capabilities can be found on: // https://www.browserstack.com/automate/capabilities?tag=selenium-4 desiredCapabilities: { "bstack:options": { local: "false", userName: "${BROWSERSTACK_USER}", accessKey: "${BROWSERSTACK_KEY}" } }, test_runner: { type: "mocha" }, disable_error_log: true, webdriver: { keep_alive: true, start_process: false } }, "browserstack.chrome": { extends: "browserstack", desiredCapabilities: { browserName: "chrome", chromeOptions: { // This tells Chromedriver to run using the legacy JSONWire protocol // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/ w3c: false } } }, "browserstack.firefox": { extends: "browserstack", desiredCapabilities: { browserName: "firefox" } }, "browserstack.ie": { extends: "browserstack", desiredCapabilities: { browserName: "IE", browserVersion: "11.0", "bstack:options": { os: "Windows", osVersion: "10", local: "false", seleniumVersion: "3.5.2", resolution: "1366x768" } } }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using the Selenium service, either locally or remote, | // like Selenium Grid | ////////////////////////////////////////////////////////////////////////////////// selenium: { // Selenium Server is running locally and is managed by Nightwatch selenium: { start_process: true, port: 4444, server_path: Services.seleniumServer ? Services.seleniumServer.path : "", cli_args: { "webdriver.gecko.driver": Services.geckodriver ? Services.geckodriver.path : "", "webdriver.chrome.driver": Services.chromedriver ? Services.chromedriver.path : "" } } }, "selenium.chrome": { extends: "selenium", desiredCapabilities: { browserName: "chrome", chromeOptions: { w3c: false } } }, "selenium.firefox": { extends: "selenium", desiredCapabilities: { browserName: "firefox", "moz:firefoxOptions": { args: [ // '-headless', // '-verbose' ] } } } } }; function loadServices() { try { Services.seleniumServer = require("selenium-server"); } catch (err) {} try { Services.chromedriver = require("chromedriver"); } catch (err) {} try { Services.geckodriver = require("geckodriver"); } catch (err) {} } ```

Your Environment

Executable Version
nightwatch --version nightwatch v1.3.4
npm --version 6.14.2
yarn --version n/a
node --version v10.15.0
Browser driver Version
chromedriver 80.0.1
geckodriver 1.19.1
OS Version
Windows 10 1903
aviatrix commented 4 years ago

We are also experiencing this issue, and it's supper annoying.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any recent activity. If possible, please retry using the latest Nightwatch version and update the issue with any relevant details. If no further activity occurs, it will be closed. Thank you for your contribution.