a failing async test causes any browser object calls to fail in the afterEach step

chris-jackson-actionqa commented 2 years ago

Describe the bug

When using async for the test, the afterEach hook can hang if that tests fails. It only hangs if you invoke the browser object. In my example below, I'm just pausing for one second.

If you remove the async from the test, the afterEach hook runs fine. If you remove the browser.pause call from the afterEach, it also runs fine. If the test passes, then it runs fine.

When running the test, you'll see that the console.log('Made it here') never gets invoked. Then the afterEach call will time out.

Sample test

sampleTest.js

```js // Please add the sample test here describe('repro afterEach', () => { afterEach(async (browser, done) => { await browser.pause(1_000); console.log('made it here'); browser.end(done); }); it('fail on purpose', async (browser) => { browser.url('http://example.com/'); browser.expect.element('body > div > h1').text.to.contain('blah'); }); }); ```

Run with command

$ nightwatch test/sampleTest.js -e chrome

Verbose output

debug.log

```txt ❯ npx nightwatch -e chrome --verbose [repro afterEach] Test Suite ──────────────────────────────────────────────────────── ⠋ Starting ChromeDriver on port 9515... Starting ChromeDriver with server_path=/Users/cjackson/Documents/nwrepro/node_modules/chromedriver/lib/chromedriver/chromedriver... Request POST /session { desiredCapabilities: { browserName: 'chrome', 'goog:chromeOptions': { w3c: true, args: [] }, name: 'repro afterEach' }, capabilities: { alwaysMatch: { browserName: 'chrome', 'goog:chromeOptions': { w3c: true, args: [] } } } ⠦ Starting ChromeDriver on port 9515... Response 200 POST /session (1316ms) { value: { capabilities: { acceptInsecureCerts: false, browserName: 'chrome', browserVersion: '101.0.4951.64', chrome: { chromedriverVersion: '101.0.4951.41 (93c720db8323b3ec10d056025ab95c23a31997c9-refs/branch-heads/4951@{#904})', userDataDir: '/var/folders/91/l15sfgjs7cg2d0htqwh6h93w0000gn/T/.com.google.Chrome.uQRJ6L' }, 'goog:chromeOptions': { debuggerAddress: 'localhost:52475' }, networkConnectionEnabled: false, pageLoadStrategy: 'normal', platformName: 'mac os x', proxy: {}, setWindowRect: true, strictFileInteractability: false, timeouts: { implicit: 0, pageLoad: 300000, script: 30000 }, unhandledPromptBehavior: 'dismiss and notify', 'webauthn:extension:credBlob': true, 'webauthn:extension:largeBlob': true, 'webauthn:virtualAuthenticators': true }, sessionId: '370fe688bdb4bc37e3c0514f4418a4a3' } ℹ Connected to ChromeDriver on port 9515 (1383ms). Using: chrome (101.0.4951.64) on MAC OS X. Received session with ID: 370fe688bdb4bc37e3c0514f4418a4a3 → Running [before]: → Completed [before]. Running fail on purpose: ─────────────────────────────────────────────────────────────────────────────────────────────────── → Running [beforeEach]: → Completed [beforeEach]. → Running [afterEach]: → Running command: url ('http://example.com/') ⠋ Loading url: http://example.com/ Request POST /session/370fe688bdb4bc37e3c0514f4418a4a3/url ⠴ Loading url: http://example.com/ Response 200 POST /session/370fe688bdb4bc37e3c0514f4418a4a3/url (415ms) ℹ Loaded url http://example.com/ in 416ms → Completed command: url ('http://example.com/') (417ms) → Running command: expect.element ('body > div > h1') Request POST /session/370fe688bdb4bc37e3c0514f4418a4a3/elements { using: 'css selector', value: 'body > div > h1' } Response 200 POST /session/370fe688bdb4bc37e3c0514f4418a4a3/elements (16ms) { value: [ { 'element-6066-11e4-a52e-4f735466cecf': '10746a0c-62f9-45b2-9464-bff810997d66' } ] } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (19ms) { value: 'Example Domain' } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms) { value: 'Example Domain' } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms) { value: 'Example Domain' } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (6ms) { value: 'Example Domain' } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms) { value: 'Example Domain' } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (9ms) { value: 'Example Domain' } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms) { value: 'Example Domain' } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (7ms) { value: 'Example Domain' } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (7ms) { value: 'Example Domain' } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms) { value: 'Example Domain' } Request GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text Response 200 GET /session/370fe688bdb4bc37e3c0514f4418a4a3/element/10746a0c-62f9-45b2-9464-bff810997d66/text (8ms) { value: 'Example Domain' } ✖ Expected element div > h1> text to contain: "blah" - expected "contain 'blah'" but got: "Example Domain" (5152ms) at DescribeInstance. (/Users/cjackson/Documents/nwrepro/tests/reproAfterEach.js:9:20) → Completed command: expect.element ('body > div > h1') (5153ms) TimeoutError: done() callback timeout of 20000ms was reached while executing "afterEach". Make sure to call the done() callback when the operation finishes. → Running [after]: → Completed [after]. → Running command: end () → Running command: session ('delete', [Function]) Request DELETE /session/370fe688bdb4bc37e3c0514f4418a4a3 Response 200 DELETE /session/370fe688bdb4bc37e3c0514f4418a4a3 (53ms) { value: null } → Completed command: session ('delete', [Function]) (54ms) Wrote log file to: /Users/cjackson/Documents/nwrepro/logs/reproAfterEach_chromedriver.log. → Completed command: end () (57ms) ────────────────────────────────────────────────────────────────────────────────────────────────────────────────── TEST FAILURE (21.57s): - 1 error during execution; - 1 assertions failed; 0 passed TimeoutError: done() callback timeout of 20000ms was reached while executing "afterEach". Make sure to call the done() callback when the operation finishes. ✖ 1) reproAfterEach – fail on purpose Expected element div > h1> text to contain: "blah" - expected "contain 'blah'" but got: "Example Domain" (5152ms) at DescribeInstance. (/Users/cjackson/Documents/nwrepro/tests/reproAfterEach.js:9:20) ChromeDriver process closed. Wrote report file to: tests_output/CHROME_101.0.4951.64__reproAfterEach.xml. ```

Configuration

nightwatch.json

```js // // Refer to the online docs for more details: // https://nightwatchjs.org/gettingstarted/configuration/ // // _ _ _ _ _ _ _ // | \ | |(_) | | | | | | | | // | \| | _ __ _ | |__ | |_ __ __ __ _ | |_ ___ | |__ // | . ` || | / _` || '_ \ | __|\ \ /\ / / / _` || __| / __|| '_ \ // | |\ || || (_| || | | || |_ \ V V / | (_| || |_ | (__ | | | | // \_| \_/|_| \__, ||_| |_| \__| \_/\_/ \__,_| \__| \___||_| |_| // __/ | // |___/ // module.exports = { // An array of folders (excluding subfolders) where your tests are located; // if this is not specified, the test source must be passed as the second argument to the test runner. src_folders: ['tests'], // See https://nightwatchjs.org/guide/working-with-page-objects/using-page-objects.html page_objects_path: ['node_modules/nightwatch/examples/pages/'], // See https://nightwatchjs.org/guide/extending-nightwatch/custom-commands.html custom_commands_path: ['node_modules/nightwatch/examples/custom-commands/'], // See https://nightwatchjs.org/guide/extending-nightwatch/custom-assertions.html custom_assertions_path: '', // See https://nightwatchjs.org/guide/extending-nightwatch/plugin-api.html plugins: [], // See https://nightwatchjs.org/guide/#external-globals globals_path: '', webdriver: {}, test_settings: { default: { disable_error_log: false, launch_url: 'https://nightwatchjs.org', screenshots: { enabled: false, path: 'screens', on_failure: true, }, desiredCapabilities: { browserName: 'firefox', }, webdriver: { start_process: true, server_path: '', }, }, safari: { desiredCapabilities: { browserName: 'safari', alwaysMatch: { acceptInsecureCerts: false, }, }, webdriver: { start_process: true, server_path: '', }, }, firefox: { desiredCapabilities: { browserName: 'firefox', alwaysMatch: { acceptInsecureCerts: true, 'moz:firefoxOptions': { args: [ // '-headless', // '-verbose' ], }, }, }, webdriver: { start_process: true, server_path: '', cli_args: [ // very verbose geckodriver logs // '-vv' ], }, }, chrome: { desiredCapabilities: { browserName: 'chrome', 'goog:chromeOptions': { // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/ // // w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78) w3c: true, args: [ //'--no-sandbox', //'--ignore-certificate-errors', //'--allow-insecure-localhost', //'--headless' ], }, }, webdriver: { start_process: true, server_path: '', cli_args: [ // --verbose ], }, }, edge: { desiredCapabilities: { browserName: 'MicrosoftEdge', 'ms:edgeOptions': { w3c: true, // More info on EdgeDriver: https://docs.microsoft.com/en-us/microsoft-edge/webdriver-chromium/capabilities-edge-options args: [ //'--headless' ], }, }, webdriver: { start_process: true, // Download msedgedriver from https://docs.microsoft.com/en-us/microsoft-edge/webdriver-chromium/ // and set the location below: server_path: '', cli_args: [ // --verbose ], }, }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using cucumber-js (https://cucumber.io) | // | // It uses the bundled examples inside the nightwatch examples folder; feel free | // to adapt this to your own project needs | ////////////////////////////////////////////////////////////////////////////////// 'cucumber-js': { src_folders: ['examples/cucumber-js/features/step_definitions'], test_runner: { // set cucumber as the runner type: 'cucumber', // define cucumber specific options options: { //set the feature path feature_path: 'node_modules/nightwatch/examples/cucumber-js/*/*.feature', // start the webdriver session automatically (enabled by default) // auto_start_session: true // use parallel execution in Cucumber // parallel: 2 // set number of workers to use (can also be defined in the cli as --parallel 2 }, }, }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using the browserstack.com cloud service | // | // Please set the username and access key by setting the environment variables: | // - BROWSERSTACK_USERNAME | // - BROWSERSTACK_ACCESS_KEY | // .env files are supported | ////////////////////////////////////////////////////////////////////////////////// browserstack: { selenium: { host: 'hub.browserstack.com', port: 443, }, // More info on configuring capabilities can be found on: // https://www.browserstack.com/automate/capabilities?tag=selenium-4 desiredCapabilities: { 'bstack:options': { userName: '${BROWSERSTACK_USERNAME}', accessKey: '${BROWSERSTACK_ACCESS_KEY}', }, }, disable_error_log: true, webdriver: { timeout_options: { timeout: 15000, retry_attempts: 3, }, keep_alive: true, start_process: false, }, }, 'browserstack.local': { extends: 'browserstack', desiredCapabilities: { 'browserstack.local': true, }, }, 'browserstack.chrome': { extends: 'browserstack', desiredCapabilities: { browserName: 'chrome', chromeOptions: { w3c: true, }, }, }, 'browserstack.firefox': { extends: 'browserstack', desiredCapabilities: { browserName: 'firefox', }, }, 'browserstack.ie': { extends: 'browserstack', desiredCapabilities: { browserName: 'internet explorer', browserVersion: '11.0', }, }, 'browserstack.safari': { extends: 'browserstack', desiredCapabilities: { browserName: 'safari', }, }, 'browserstack.local_chrome': { extends: 'browserstack.local', desiredCapabilities: { browserName: 'chrome', }, }, 'browserstack.local_firefox': { extends: 'browserstack.local', desiredCapabilities: { browserName: 'firefox', }, }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using the SauceLabs cloud service | // | // Please set the username and access key by setting the environment variables: | // - SAUCE_USERNAME | // - SAUCE_ACCESS_KEY | ////////////////////////////////////////////////////////////////////////////////// saucelabs: { selenium: { host: 'ondemand.saucelabs.com', port: 443, }, // More info on configuring capabilities can be found on: // https://wiki.saucelabs.com/display/DOCS/Test+Configuration+Options desiredCapabilities: { 'sauce:options': { username: '${SAUCE_USERNAME}', accessKey: '${SAUCE_ACCESS_KEY}', // https://docs.saucelabs.com/dev/cli/sauce-connect-proxy/#--region // region: 'us-west-1' // https://docs.saucelabs.com/dev/test-configuration-options/#tunnelidentifier // parentTunnel: '', // tunnelIdentifier: '', }, }, disable_error_log: false, webdriver: { start_process: false, }, }, 'saucelabs.chrome': { extends: 'saucelabs', desiredCapabilities: { browserName: 'chrome', screenResolution: '1280x1024', browserVersion: 'latest', javascriptEnabled: true, acceptSslCerts: true, timeZone: 'London', chromeOptions: { w3c: true, }, }, }, 'saucelabs.firefox': { extends: 'saucelabs', desiredCapabilities: { browserName: 'firefox', screenResolution: '1280x1024', browserVersion: 'latest', javascriptEnabled: true, acceptSslCerts: true, timeZone: 'London', }, }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using the Selenium service, either locally or remote, | // like Selenium Grid | ////////////////////////////////////////////////////////////////////////////////// selenium_server: { // Selenium Server is running locally and is managed by Nightwatch // Install the NPM package @nightwatch/selenium-server or download the selenium server jar file from https://github.com/SeleniumHQ/selenium/releases/, e.g.: selenium-server-4.1.1.jar selenium: { start_process: true, port: 4444, server_path: '', // Leave empty if @nightwatch/selenium-server is installed command: 'standalone', // Selenium 4 only cli_args: { //'webdriver.gecko.driver': '', //'webdriver.chrome.driver': '' }, }, webdriver: { start_process: false, default_path_prefix: '/wd/hub', }, }, 'selenium.chrome': { extends: 'selenium_server', desiredCapabilities: { browserName: 'chrome', chromeOptions: { w3c: true, }, }, }, 'selenium.firefox': { extends: 'selenium_server', desiredCapabilities: { browserName: 'firefox', 'moz:firefoxOptions': { args: [ // '-headless', // '-verbose' ], }, }, }, }, }; ```

Your Environment

Executable	Version
`nightwatch --version`	2.15
`npm --version`	8.1.2
`yarn --version`	1.22.17
`node --version`	v16.13.1

Browser driver	Version
chromedriver	101.0.4951.41

OS	Version
macOS Big Sur	11.6.5

AutomatedTester commented 2 years ago

To be tested with https://github.com/nightwatchjs/nightwatch/pull/3328

gravityvi commented 2 years ago

Removing async will fix the issue as I can see it's not required. You can either return a promise or use await both will resolve the issue.

  it('fail on purpose',  async function() {
    browser.url('http://example.com/');
    await browser.expect.element('body > div > h1').text.to.contain('blah');
  });

  it('fail on purpose', async function() {
    browser.url('http://example.com/');
    return browser.expect.element('body > div > h1').text.to.contain('blah');
  });

nightwatchjs / nightwatch