Emit in custom command in before/beforeEach does not fail

lloiser commented 4 years ago

Describe the bug

Running a custom command that emits an error in the before/beforeEach hook function does not fail the test. (note: this probably also affects after/afterEach but I have not tested it)

Sample

Given a command that always fails after 100ms:

commands/emitError.js

```js const Events = require('events'); module.exports = class CustomPause extends Events { command() { setTimeout(() => { console.log(">>>>> Emitting error...") this.emit("error", new Error("FAIL!")); console.log(">>>>> Emitted error!") }, 100); return this; } }; ```

When I call this command in the before function it ignores the error and simply continues with the test (that eventually succeeds):

test/emitError-before.js

```js module.exports = { before(client) { client.init(); client.emitError(); // <-- should fail here but does not! }, myTestFunction(client) { client.waitForElementVisible("body") }, after(client) { client.end(); } }; ```

output

```txt $ npx nightwatch ./test/emitError-before.js --env chrome [Emit Error Before] Test Suite ============================== ℹ Connected to localhost on port 9515 (982ms). Using: chrome (79.0.3945.117) on Mac OS X platform. >>>>> Emitting error... >>>>> Emitted error! Running: myTestFunction ✔ Element was visible after 26 milliseconds. OK. 1 assertions passed. (32ms) ```

When I call this command not in the before function it will end the test.

test/emitError-notBefore.js

```js module.exports = { notBefore(client) { client.init(); client.emitError(); // <-- actually fails here }, myTestFunction(client) { client.waitForElementVisible("body") }, after(client) { client.end(); } }; ```

output

```txt $ npx nightwatch ./test/emitError-notBefore.js --env chrome [Emit Error Not Before] Test Suite ================================== ℹ Connected to localhost on port 9515 (886ms). Using: chrome (79.0.3945.117) on Mac OS X platform. Running: notBefore >>>>> Emitting error... >>>>> Emitted error! FAILED: 1 errors (1.588s) Error: Error while running "emitError" command: FAIL! at Timeout.setTimeout [as _onTimeout] (/nightwatch-issue/commands/emitError.js:7:32) _________________________________________________ TEST FAILURE: 1 error during execution; 0 tests failed, 0 passed (2.718s) ✖ emitError-notBefore – notBefore (1.588s) Error: Error while running "emitError" command: FAIL! at Timeout.setTimeout [as _onTimeout] (/nightwatch-issue/commands/emitError.js:7:32) SKIPPED: - myTestFunction ```

For comparison, when I run a standard command (like waitForElementVisible) in the before function it will in the end fail the test (which is good) but continues with the rest of the test functions which is unexpected.

test/standardCommand-before.js

```js module.exports = { before(client) { client.init("https://nightwatchjs.org"); client.waitForElementVisible('#abc'); // <-- should fail here! }, myTestFunction(client) { client.waitForElementVisible("body") }, after(client) { client.end(); } }; ```

output

```txt $ npx nightwatch ./test/standardCommand-before.js --env chrome [Standard Command Before] Test Suite ==================================== ℹ Connected to localhost on port 9515 (1008ms). Using: chrome (79.0.3945.117) on Mac OS X platform. ✖ Timed out while waiting for element <#abc> to be present for 5000 milliseconds. - expected "visible" but got: "not found" (5123ms) at Object.before (/nightwatch-issue/test/standardCommand-before.js:5:16) at new Promise () Running: myTestFunction ✔ Element was visible after 15 milliseconds. OK. 1 assertions passed. (17ms) _________________________________________________ TEST FAILURE: 1 assertions failed, 1 passed (8.237s) ✖ standardCommand-before ```

Verbose output of the failing `test/emitError-before.js` test

debug.log

```txt $ npx nightwatch ./test/emitError-before.js --env chrome --verbose Starting ChromeDriver on port 9515... ChromeDriver up and running on port 9515 with pid: 12137 (130ms). [Emit Error Before] Test Suite ============================== ⠋ Connecting to localhost on port 9515... Request POST /session { desiredCapabilities: { browserName: 'chrome', platform: 'ANY', chromeOptions: { args: [] }, ⠙ Connecting to localhost on port 9515... Response 200 POST /session (943ms) { sessionId: '2419b1e79a407ae077113a8fbceea43e', status: 0, value: { acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: 'chrome', chrome: { chromedriverVersion: '79.0.3945.36 (3582db32b33893869b8c1339e8f4d9ed1816f143-refs/branch-heads/3945@{#614})', userDataDir: '/var/folders/g2/hj8cnxyj3rq8260dwm6672k00000gn/T/.com.google.Chrome.IbvwaS' }, cssSelectorsEnabled: true, databaseEnabled: false, 'goog:chromeOptions': { debuggerAddress: 'localhost:60858' }, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: 'normal', platform: 'Mac OS X', proxy: {}, rotatable: false, setWindowRect: true, strictFileInteractability: false, takesHeapSnapshot: true, takesScreenshot: true, timeouts: { implicit: 0, pageLoad: 300000, script: 30000 }, unexpectedAlertBehaviour: 'ignore', version: '79.0.3945.117', ℹ Connected to localhost on port 9515 (986ms). Using: chrome (79.0.3945.117) on Mac OS X platform. Received session with ID: 2419b1e79a407ae077113a8fbceea43e → Running [before]: → Running command: init ('https://nightwatchjs.org') → Running command: url ('https://nightwatchjs.org', [Function]) Request POST /session/2419b1e79a407ae077113a8fbceea43e/url { url: 'https://nightwatchjs.org' } Response 200 POST /session/2419b1e79a407ae077113a8fbceea43e/url (1446ms) { sessionId: '2419b1e79a407ae077113a8fbceea43e', status: 0, value: null } → Completed command: init ('https://nightwatchjs.org') (1450ms) → Completed command: url ('https://nightwatchjs.org', [Function]) (1448ms) → Running command: emitError () >>>>> Emitting error... >>>>> Emitted error! → Completed command: emitError () (104ms) → Completed [before]. Running: myTestFunction → Running [beforeEach]: → Completed [beforeEach]. → Running command: waitForElementVisible ('body') Request POST /session/2419b1e79a407ae077113a8fbceea43e/elements { using: 'css selector', value: 'body' } Response 200 POST /session/2419b1e79a407ae077113a8fbceea43e/elements (13ms) { sessionId: '2419b1e79a407ae077113a8fbceea43e', status: 0, value: [ { ELEMENT: '0.886048504250488-1' } ] } Request GET /session/2419b1e79a407ae077113a8fbceea43e/element/0.886048504250488-1/displayed Response 200 GET /session/2419b1e79a407ae077113a8fbceea43e/element/0.886048504250488-1/displayed (8ms) { sessionId: '2419b1e79a407ae077113a8fbceea43e', status: 0, value: true } ✔ Element was visible after 24 milliseconds. → Completed command: waitForElementVisible ('body') (26ms) → Running [afterEach]: → Completed [afterEach]. OK. 1 assertions passed. (28ms) → Running [after]: → Running command: end () → Running command: session ('delete', [Function]) Request DELETE /session/2419b1e79a407ae077113a8fbceea43e Response 200 DELETE /session/2419b1e79a407ae077113a8fbceea43e (51ms) { sessionId: '2419b1e79a407ae077113a8fbceea43e', status: 0, value: null } → Completed command: end () (53ms) → Completed command: session ('delete', [Function]) (53ms) → Completed [after]. Wrote report file to: tests_output/CHROME_79.0.3945.117_Mac_OS_X_emitError-before.xml. Wrote log file to: /nightwatch-issue/chromedriver.log. ChromeDriver process closed. ```

Configuration

I have used the automatically generated configuration, added the src_folders and custom_commands_path and removed the unnecessary environments:

nightwatch.json

```js // Autogenerated by Nightwatch // Refer to the online docs for more details: https://nightwatchjs.org/gettingstarted/configuration/ const Services = {}; loadServices(); module.exports = { // An array of folders (excluding subfolders) where your tests are located; // if this is not specified, the test source must be passed as the second argument to the test runner. src_folders: ["tests"], // See https://nightwatchjs.org/guide/working-with-page-objects/ page_objects_path: '', // See https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-commands custom_commands_path: 'commands', // See https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-assertions custom_assertions_path: '', // See https://nightwatchjs.org/guide/#external-globals globals_path: '', webdriver: {}, test_settings: { default: { disable_error_log: false, launch_url: 'https://nightwatchjs.org', screenshots: { enabled: false, path: 'screens', on_failure: true }, desiredCapabilities: { browserName: 'firefox' }, webdriver: { start_process: true, server_path: (Services.geckodriver ? Services.geckodriver.path : '') } }, chrome: { desiredCapabilities: { browserName: 'chrome', chromeOptions: { // This tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78) // w3c: false, // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/ args: [ //'--no-sandbox', //'--ignore-certificate-errors', //'--allow-insecure-localhost', //'--headless' ] } }, webdriver: { start_process: true, port: 9515, server_path: (Services.chromedriver ? Services.chromedriver.path : ''), cli_args: [ // --verbose ] } } } }; function loadServices() { try { Services.seleniumServer = require('selenium-server'); } catch (err) { } try { Services.chromedriver = require('chromedriver'); } catch (err) { } try { Services.geckodriver = require('geckodriver'); } catch (err) { } } ```

Your Environment

Executable	Version
`nightwatch --version`	nightwatch v1.3.2
`npm --version`	6.11.3
`node --version`	v10.16.3

Browser driver	Version
chromedriver	79

OS	Version
macOS Catalina	10.15.2

beatfactor commented 4 years ago

what happens if you throw the error instead?

lloiser commented 4 years ago

This seems to work. It stops in the before and does not continue with my actual test function.

$ npx nightwatch ./test/emitError-before.js --env chrome 

[Emit Error Before] Test Suite
==============================
ℹ Connected to localhost on port 9515 (1040ms).
  Using: chrome (79.0.3945.117) on Mac OS X platform.

>>>>> Emitting error...
_________________________________________________

TEST FAILURE: 1 error during execution; 0 tests failed, 0 passed (2.785s)

  Error: FAIL!
       at Timeout.setTimeout [as _onTimeout] (/nightwatch-issue/emit-error/commands/emitError.js:7:19)

   Error: FAIL!
       at Timeout.setTimeout [as _onTimeout] (/nightwatch-issue/emit-error/commands/emitError.js:7:19)

Still this is a "fix" for the custom commands only. Standard commands (waitForElementVisible in my test above) from nightwatch still continue with the test, but report the failure afterwards.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any recent activity. If possible, please retry using the latest Nightwatch version and update the issue with any relevant details. If no further activity occurs, it will be closed. Thank you for your contribution.

nightwatchjs / nightwatch