search

nightwatchjs / nightwatch

Integrated end-to-end testing framework written in Node.js and using W3C Webdriver API. Developed at @browserstack
https://nightwatchjs.org
MIT License
11.83k stars 1.32k forks source link

Nightwatch 2.0 - async custom command pauses in before each but not in test body #2980

Closed kelly-tock closed 1 year ago

kelly-tock commented 2 years ago

Describe the bug

a custom command that is async and sets a cookie with a pause behaves as expected when in beforeEach, but not when in the test body.

Custom command

module.exports = class ExampleWithPause {
  async command() {
    await this.api.url('https://exampledomainthatworks.com/blank.html');
    await this.api.setCookie({
      name: 'TEST',
      value: 'random',
      domain: '.exampledomainthatworks.com',
    });
    await this.api.pause(30000);
  }
};

Sample test

sampleTest.js

```js // Please add the sample test here describe('login', function () { beforeEach(async function (browser) { // pauses correctly if (browser.launchUrl.startsWith('https://example.com')) { await browser.exampleWithPause(); } }); it('logs in and displays the homepage', async function (browser) { const login = browser.page.loginPage(); const global = browser.page.global(); await login .navigate() .setValue('@emailInput', 'example@example.com') .setValue('@passwordInput', 'example') .click('@submit'); const body = global.elements.body; browser.expect.element(body).text.to.contain('Expected today'); browser.end(); }); }); ``` ```js describe('login', function () { // beforeEach(async function (browser) { // if (browser.launchUrl.startsWith('https://staging.com')) { // await browser.skipBasicAuth(); // } // }); it('logs in and displays the homepage', async function (browser) { const login = browser.page.loginPage(); const global = browser.page.global(); if (browser.launchUrl.startsWith('https://staging')) { // does not pause await browser.exampleWithPause(); } await login .navigate() .setValue('@emailInput', 'test@example.com') .setValue('@passwordInput', 'example') .click('@submit'); const body = global.elements.body; browser.expect.element(body).text.to.contain('Expected today'); browser.end(); }); }); ```

Run with command

$ nightwatch test/sampleTest.js --your-other-arguments-here

Verbose output

debug.log

```txt ⠋ Starting ChromeDriver on port 9515... Starting ChromeDriver with server_path=/Users/kelly/dashboard/node_modules/chromedriver/lib/chromedriver/chromedriver... Request POST /session { desiredCapabilities: { browserName: 'chrome', name: 'login', 'goog:chromeOptions': {} }, capabilities: { alwaysMatch: { browserName: 'chrome', 'goog:chromeOptions': {} } } ⠏ Starting ChromeDriver on port 9515... Response 200 POST /session (3122ms) { value: { capabilities: { acceptInsecureCerts: false, browserName: 'chrome', browserVersion: '96.0.4664.110', chrome: { chromedriverVersion: '96.0.4664.45 (76e4c1bb2ab4671b8beba3444e61c0f17584b2fc-refs/branch-heads/4664@{#947})', userDataDir: '/var/folders/qj/rtpr5f951xx6lr_zsxvc09_w0000gn/T/.com.google.Chrome.cfwMOu' }, 'goog:chromeOptions': { debuggerAddress: 'localhost:62966' }, networkConnectionEnabled: false, pageLoadStrategy: 'normal', platformName: 'mac os x', proxy: {}, setWindowRect: true, strictFileInteractability: false, timeouts: { implicit: 0, pageLoad: 300000, script: 30000 }, unhandledPromptBehavior: 'dismiss and notify', 'webauthn:extension:credBlob': true, 'webauthn:extension:largeBlob': true, 'webauthn:virtualAuthenticators': true }, sessionId: '8033eaf4f86081027dd75109c29144d8' } ℹ Connected to ChromeDriver on port 9515 (3200ms). Using: chrome (96.0.4664.110) on MAC OS X. Received session with ID: 8033eaf4f86081027dd75109c29144d8 → Running [before]: → Completed [before]. Running logs in and displays the hub: ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── → Running [beforeEach]: → Completed [beforeEach]. → Running command: skipBasicAuth () → Running command: url ('https://staging/test.html') Request POST /session/8033eaf4f86081027dd75109c29144d8/url { url: 'https://staging/test.html' } Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/url (542ms) { value: null } → Completed command: url ('https://staging/test.html') (545ms) → Running command: setCookie ({name, value, domain}) → Running command: cookie ('POST', {name, value, domain}, [Function]) Request POST /session/8033eaf4f86081027dd75109c29144d8/cookie { cookie: { name: 'TEST', value: 'random', path: undefined, domain: '.domainexample.com', secure: false, httpOnly: false, expiry: undefined, sameSite: undefined } } Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/cookie (34ms) { value: null } → Completed command: setCookie ({name, value, domain}) (35ms) → Running command: url ('https://staginglogin', ) → Completed command: cookie ('POST', {name, value, domain}, [Function]) (35ms) → Running command: pause (30000) Request POST /session/8033eaf4f86081027dd75109c29144d8/url { url: 'https://staging/login' } Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/url (5432ms) { value: null } → Completed command: url ('https://staging/login', ) (5434ms) → Running command: setValue ({name, __index, __selector, locateStrategy, pseudoSelector, parent, resolvedElement, abortOnFailure, suppressNotFoundErrors, timeout, retryInterval, message}, 'test+nightwatchadmin@tockhq.com') → Completed command: skipBasicAuth () (6019ms) Request POST /session/8033eaf4f86081027dd75109c29144d8/elements { using: 'css selector', value: '[data-testid="login-email-input"]' } Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/elements (32ms) { value: [ { 'element-6066-11e4-a52e-4f735466cecf': '6f4ac005-334f-4bc9-9857-ab9ddd5e73a5' } ] } Request POST /session/8033eaf4f86081027dd75109c29144d8/element/6f4ac005-334f-4bc9-9857-ab9ddd5e73a5/clear {} Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/element/6f4ac005-334f-4bc9-9857-ab9ddd5e73a5/clear (45ms) { value: null } Request POST /session/8033eaf4f86081027dd75109c29144d8/element/6f4ac005-334f-4bc9-9857-ab9ddd5e73a5/value { text: 'test@test.com', value: [ ] } Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/element/6f4ac005-334f-4bc9-9857-ab9ddd5e73a5/value (1929ms) { value: null } → Completed command: setValue ({name, __index, __selector, locateStrategy, pseudoSelector, parent, resolvedElement, abortOnFailure, suppressNotFoundErrors, timeout, retryInterval, message}, 'test+nightwatchadmin@tockhq.com') (2012ms) → Running command: setValue ({name, __index, __selector, locateStrategy, pseudoSelector, parent, resolvedElement, abortOnFailure, suppressNotFoundErrors, timeout, retryInterval, message}, 'password1') Request POST /session/8033eaf4f86081027dd75109c29144d8/elements { using: 'css selector', value: '[data-testid="login-password-input"]' } Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/elements (169ms) { value: [ { 'element-6066-11e4-a52e-4f735466cecf': '7b1e27e0-286f-4cd7-8503-d2dad4a52395' } ] } Request POST /session/8033eaf4f86081027dd75109c29144d8/element/7b1e27e0-286f-4cd7-8503-d2dad4a52395/clear {} Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/element/7b1e27e0-286f-4cd7-8503-d2dad4a52395/clear (244ms) { value: null } Request POST /session/8033eaf4f86081027dd75109c29144d8/element/7b1e27e0-286f-4cd7-8503-d2dad4a52395/value { text: 'password1', value: [ ] } Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/element/7b1e27e0-286f-4cd7-8503-d2dad4a52395/value (37ms) { value: null } → Completed command: setValue ({name, __index, __selector, locateStrategy, pseudoSelector, parent, resolvedElement, abortOnFailure, suppressNotFoundErrors, timeout, retryInterval, message}, 'password1') (451ms) → Running command: click ({name, __index, __selector, locateStrategy, pseudoSelector, parent, resolvedElement, abortOnFailure, suppressNotFoundErrors, timeout, retryInterval, message}) Request POST /session/8033eaf4f86081027dd75109c29144d8/elements { using: 'css selector', value: '[data-testid="login-submit-button"]' } Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/elements (10ms) { value: [ { 'element-6066-11e4-a52e-4f735466cecf': '6e42f579-3528-41d7-8d64-46b57ab696ce' } ] } Request POST /session/8033eaf4f86081027dd75109c29144d8/element/6e42f579-3528-41d7-8d64-46b57ab696ce/click {} Response 200 POST /session/8033eaf4f86081027dd75109c29144d8/element/6e42f579-3528-41d7-8d64-46b57ab696ce/click (26ms) { value: null } → Completed command: click ({name, __index, __selector, locateStrategy, pseudoSelector, parent, resolvedElement, abortOnFailure, suppressNotFoundErrors, timeout, retryInterval, message}) (36ms) → Running [afterEach]: → Completed [afterEach]. No assertions ran. → Running [after]: → Completed [after]. ```

Configuration

nightwatch.json

```js // Autogenerated by Nightwatch // Refer to the online docs for more details: https://nightwatchjs.org/gettingstarted/configuration/ const Services = {}; loadServices(); // _ _ _ _ _ _ _ // | \ | |(_) | | | | | | | | // | \| | _ __ _ | |__ | |_ __ __ __ _ | |_ ___ | |__ // | . ` || | / _` || '_ \ | __|\ \ /\ / / / _` || __| / __|| '_ \ // | |\ || || (_| || | | || |_ \ V V / | (_| || |_ | (__ | | | | // \_| \_/|_| \__, ||_| |_| \__| \_/\_/ \__,_| \__| \___||_| |_| // __/ | // |___/ module.exports = { // An array of folders (excluding subfolders) where your tests are located; // if this is not specified, the test source must be passed as the second argument to the test runner. src_folders: ['e2e/tests/'], // See https://nightwatchjs.org/guide/working-with-page-objects/ page_objects_path: ['e2e/pages/'], // See https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-commands custom_commands_path: ['e2e/custom-commands/'], // See https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-assertions custom_assertions_path: '', // See https://nightwatchjs.org/guide/#external-globals globals_path: '', globals: { retryAssertionTimeout: 15000, waitForConditionTimeout: 15000, }, webdriver: {}, test_settings: { default: { disable_error_log: false, launch_url: '${LAUNCH_URL}', screenshots: { enabled: true, path: 'screens', on_failure: true, }, desiredCapabilities: { browserName: 'chrome', }, webdriver: { start_process: true, server_path: '', }, }, safari: { desiredCapabilities: { browserName: 'safari', alwaysMatch: { acceptInsecureCerts: false, }, }, webdriver: { start_process: true, server_path: '', }, }, firefox: { desiredCapabilities: { browserName: 'firefox', alwaysMatch: { acceptInsecureCerts: true, 'moz:firefoxOptions': { args: [ // '-headless', // '-verbose' ], }, }, }, webdriver: { start_process: true, server_path: '', cli_args: [ // very verbose geckodriver logs // '-vv' ], }, }, chrome: { desiredCapabilities: { browserName: 'chrome', 'goog:chromeOptions': { // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/ // // w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78) w3c: true, args: [ //'--no-sandbox', //'--ignore-certificate-errors', //'--allow-insecure-localhost', //'--headless' ], }, }, webdriver: { start_process: true, server_path: '', cli_args: [ // --verbose ], timeout_options: { timeout: 20000, retry_attempts: 3, }, }, }, edge: { desiredCapabilities: { browserName: 'MicrosoftEdge', 'ms:edgeOptions': { w3c: true, // More info on EdgeDriver: https://docs.microsoft.com/en-us/microsoft-edge/webdriver-chromium/capabilities-edge-options args: [ //'--headless' ], }, }, webdriver: { start_process: true, // Download msedgedriver from https://docs.microsoft.com/en-us/microsoft-edge/webdriver-chromium/ // and set the location below: server_path: '', cli_args: [ // --verbose ], }, }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using cucumber-js (https://cucumber.io) | // | // It uses the bundled examples inside the nightwatch examples folder; feel free | // to adapt this to your own project needs | ////////////////////////////////////////////////////////////////////////////////// 'cucumber-js': { src_folders: ['examples/cucumber-js/features/step_definitions'], test_runner: { // set cucumber as the runner type: 'cucumber', // define cucumber specific options options: { //set the feature path feature_path: 'node_modules/nightwatch/examples/cucumber-js/*/*.feature', // start the webdriver session automatically (enabled by default) // auto_start_session: true // use parallel execution in Cucumber // parallel: 2 // set number of workers to use (can also be defined in the cli as --parallel 2 }, }, }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using the browserstack.com cloud service | // | // Please set the username and access key by setting the environment variables: | // - BROWSERSTACK_USER | // - BROWSERSTACK_KEY | // .env files are supported | ////////////////////////////////////////////////////////////////////////////////// browserstack: { selenium: { host: 'hub-cloud.browserstack.com', port: 443, }, // More info on configuring capabilities can be found on: // https://www.browserstack.com/automate/capabilities?tag=selenium-4 desiredCapabilities: { 'bstack:options': { userName: '${BROWSERSTACK_USER}', accessKey: '${BROWSERSTACK_KEY}', }, }, disable_error_log: true, webdriver: { timeout_options: { timeout: 15000, retry_attempts: 3, }, keep_alive: true, start_process: false, }, }, 'browserstack.local': { extends: 'browserstack', desiredCapabilities: { 'browserstack.local': true, }, }, 'browserstack.chrome': { extends: 'browserstack', desiredCapabilities: { browserName: 'chrome', chromeOptions: { w3c: true, }, }, }, 'browserstack.firefox': { extends: 'browserstack', desiredCapabilities: { browserName: 'firefox', }, }, 'browserstack.ie': { extends: 'browserstack', desiredCapabilities: { browserName: 'internet explorer', browserVersion: '11.0', }, }, 'browserstack.safari': { extends: 'browserstack', desiredCapabilities: { browserName: 'safari', }, }, 'browserstack.local_chrome': { extends: 'browserstack.local', desiredCapabilities: { browserName: 'chrome', }, }, 'browserstack.local_firefox': { extends: 'browserstack.local', desiredCapabilities: { browserName: 'firefox', }, }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using the Selenium service, either locally or remote, | // like Selenium Grid | ////////////////////////////////////////////////////////////////////////////////// selenium_server: { // Selenium Server is running locally and is managed by Nightwatch selenium: { start_process: true, port: 4444, server_path: Services.seleniumServer ? Services.seleniumServer.path : '', cli_args: { 'webdriver.gecko.driver': Services.geckodriver ? Services.geckodriver.path : '', 'webdriver.chrome.driver': Services.chromedriver ? Services.chromedriver.path : '', }, }, }, 'selenium.chrome': { extends: 'selenium_server', desiredCapabilities: { browserName: 'chrome', chromeOptions: { w3c: true, }, }, }, 'selenium.firefox': { extends: 'selenium_server', desiredCapabilities: { browserName: 'firefox', 'moz:firefoxOptions': { args: [ // '-headless', // '-verbose' ], }, }, }, }, }; function loadServices() { try { Services.seleniumServer = require('selenium-server'); } catch (err) {} try { Services.chromedriver = require('chromedriver'); } catch (err) {} try { Services.geckodriver = require('geckodriver'); } catch (err) {} } ```

Your Environment

Executable Version
nightwatch --version 2.0.0-beta.1
npm --version VERSION
yarn --version 1.22.17
node --version v14.16.0
Browser driver Version
chromedriver ^96.0.0
geckodriver ^2.0.4
OS Version
macOS Monterey 12.0.1
harshit-bs commented 2 years ago

I tested above sample test case in v2.0.4 and found that it's working perfectly for me.

gravityvi commented 2 years ago

Hey @kelly-tock, could you try using the latest Nightwatch v2 build and see if the issue still exists?

kelly-tock commented 2 years ago

yep, will give it a shot. thanks!

mieko-tock commented 2 years ago

I tried this with version 2.0.5 and it is still an issue. When the command is a part of the test body, the .pause() happens after the test is executed.

I'm running Chrome version: 98

Executable Version
nightwatch --version 2.0.5
npm --version VERSION
yarn --version 1.22.17
node --version v14.18.0
Browser driver Version
chromedriver ^97.0.0
harshit-bs commented 2 years ago

@mieko-tock With nightwatch v2.0.0-beta.1, I was able to replicate the issue. But, everything works flawlessly for me with the most recent version 2.1.0. So could you please check again with the most recent version to see whether the problem persists?

mieko-tock commented 2 years ago

I verified this with the following versions and confirmed the fix. I'm running Chrome version: 101

Executable Version
nightwatch --version 2.1.4
npm --version VERSION
yarn --version 1.22.17
node --version v16.13.2
Browser driver Version
chromedriver ^100.0.0
harshit-bs commented 1 year ago

Hey @mieko-tock @kelly-tock I'm closing this issue for now since it is fixed. Please feel free to reopen if it is still exists.