[Enhancement] Support verify and expect APIs from Nightwatch in Cucumber

[pawlakmaly]

Describe the bug

If you run you cucumber tests using cucumber runner, the results of the tests says that no issue was found even if the only assertion fails.

Sample test

feature file

```js Feature: Sample ecosia test Scenario: User should see search input after visiting ecosia page When User visits Ecosia page Then Search input should be visible ```

steps definition

```js const {Given, Then, When, Before} = require('@cucumber/cucumber'); When(/^User visits Ecosia page$/, () => { browser.navigateTo('https://ecosia.org/'); browser.waitForElementVisible('[data-test-id="cta-button"]'); }); Then(/^Search input should be visible$/, () => { browser.waitForElementVisible('[data-test-id="search-form-input"]'); browser.verify.not.visible('[data-test-id="search-form-input"]'); }); ```

Run with command

    "generate_report":"node create-html-report.js",
    "nightwatch:cucumber": "npx nightwatch --env cucumber-js --format json:./tests_output/test.json",
    "run_tests": "npm run nightwatch:cucumber && npm run generate_report"

Verbose output

debug.log


Configuration

nightwatch.json

``` // Autogenerated by Nightwatch // Refer to the online docs for more details: https://nightwatchjs.org/gettingstarted/configuration/ const Services = {}; loadServices(); // _ _ _ _ _ _ _ // | \ | |(_) | | | | | | | | // | \| | _ __ _ | |__ | |_ __ __ __ _ | |_ ___ | |__ // | . ` || | / _` || '_ \ | __|\ \ /\ / / / _` || __| / __|| '_ \ // | |\ || || (_| || | | || |_ \ V V / | (_| || |_ | (__ | | | | // \_| \_/|_| \__, ||_| |_| \__| \_/\_/ \__,_| \__| \___||_| |_| // __/ | // |___/ module.exports = { // An array of folders (excluding subfolders) where your tests are located; // if this is not specified, the test source must be passed as the second argument to the test runner. src_folders: [], // See https://nightwatchjs.org/guide/working-with-page-objects/ page_objects_path: ['node_modules/nightwatch/examples/pages/'], // See https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-commands custom_commands_path: ['node_modules/nightwatch/examples/custom-commands/'], // See https://nightwatchjs.org/guide/extending-nightwatch/#writing-custom-assertions custom_assertions_path: '', // See https://nightwatchjs.org/guide/#external-globals globals_path : '', webdriver: {}, test_settings: { default: { disable_error_log: false, screenshots: { enabled: false, path: 'screens', on_failure: true }, desiredCapabilities: { browserName : 'firefox' }, webdriver: { start_process: true, server_path: '' } }, firefox: { desiredCapabilities : { browserName : 'firefox', alwaysMatch: { acceptInsecureCerts: true, 'moz:firefoxOptions': { args: [ // '-headless', // '-verbose' ] } } }, webdriver: { start_process: true, server_path: '', cli_args: [ // very verbose geckodriver logs // '-vv' ] } }, chrome: { desiredCapabilities : { browserName : 'chrome', 'goog:chromeOptions' : { // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/ // // w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78) w3c: true, args: [ //'--no-sandbox', //'--ignore-certificate-errors', //'--allow-insecure-localhost', //'--headless' ] } }, webdriver: { start_process: true, server_path: '', cli_args: [ // --verbose ] } }, edge: { desiredCapabilities : { browserName : 'MicrosoftEdge', 'ms:edgeOptions' : { w3c: true, // More info on EdgeDriver: https://docs.microsoft.com/en-us/microsoft-edge/webdriver-chromium/capabilities-edge-options args: [ //'--headless' ] } }, webdriver: { start_process: true, // Download msedgedriver from https://docs.microsoft.com/en-us/microsoft-edge/webdriver-chromium/ // and set the location below: server_path: '', cli_args: [ // --verbose ] } }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using cucumber-js (https://cucumber.io) | // | // It uses the bundled examples inside the nightwatch examples folder; feel free | // to adapt this to your own project needs | ////////////////////////////////////////////////////////////////////////////////// 'cucumber-js': { src_folders: ['./step_definitions'], test_runner: { // set cucumber as the runner type: 'cucumber', // define cucumber specific options options: { //set the feature path feature_path: './features/*.feature', // start the webdriver session automatically (enabled by default) // auto_start_session: true // use parallel execution in Cucumber // parallel: 2 // set number of workers to use (can also be defined in the cli as --parallel 2 } } }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using the browserstack.com cloud service | // | // Please set the username and access key by setting the environment variables: | // - BROWSERSTACK_USER | // - BROWSERSTACK_KEY | // .env files are supported | ////////////////////////////////////////////////////////////////////////////////// browserstack: { selenium: { host: 'hub-cloud.browserstack.com', port: 443 }, // More info on configuring capabilities can be found on: // https://www.browserstack.com/automate/capabilities?tag=selenium-4 desiredCapabilities: { 'bstack:options' : { userName: '${BROWSERSTACK_USER}', accessKey: '${BROWSERSTACK_KEY}', } }, disable_error_log: true, webdriver: { timeout_options: { timeout: 15000, retry_attempts: 3 }, keep_alive: true, start_process: false } }, 'browserstack.local': { extends: 'browserstack', desiredCapabilities: { 'browserstack.local': true } }, 'browserstack.chrome': { extends: 'browserstack', desiredCapabilities: { browserName: 'chrome', chromeOptions : { w3c: true } } }, 'browserstack.firefox': { extends: 'browserstack', desiredCapabilities: { browserName: 'firefox' } }, 'browserstack.ie': { extends: 'browserstack', desiredCapabilities: { browserName: 'internet explorer', browserVersion: '11.0' } }, 'browserstack.safari': { extends: 'browserstack', desiredCapabilities: { browserName: 'safari' } }, 'browserstack.local_chrome': { extends: 'browserstack.local', desiredCapabilities: { browserName: 'chrome' } }, 'browserstack.local_firefox': { extends: 'browserstack.local', desiredCapabilities: { browserName: 'firefox' } }, ////////////////////////////////////////////////////////////////////////////////// // Configuration for when using the Selenium service, either locally or remote, | // like Selenium Grid | ////////////////////////////////////////////////////////////////////////////////// selenium_server: { // Selenium Server is running locally and is managed by Nightwatch selenium: { start_process: true, port: 4444, server_path: (Services.seleniumServer ? Services.seleniumServer.path : ''), cli_args: { 'webdriver.gecko.driver': (Services.geckodriver ? Services.geckodriver.path : ''), 'webdriver.chrome.driver': (Services.chromedriver ? Services.chromedriver.path : '') } } }, 'selenium.chrome': { extends: 'selenium_server', desiredCapabilities: { browserName: 'chrome', chromeOptions : { w3c: true } } }, 'selenium.firefox': { extends: 'selenium_server', desiredCapabilities: { browserName: 'firefox', 'moz:firefoxOptions': { args: [ // '-headless', // '-verbose' ] } } } } }; function loadServices() { try { Services.seleniumServer = require('selenium-server'); } catch (err) {} try { Services.chromedriver = require('chromedriver'); } catch (err) {} try { Services.geckodriver = require('geckodriver'); } catch (err) {} } ```

Your Environment

Executable	Version
nightwatch --version	2 alpha 1
npm --version	VERSION
yarn --version	VERSION
node --version	VERSION

Browser driver	Version
NAME	VERSION

OS	Version
NAME	VERSION

[pawlakmaly]

Sometimes I get the results after the tests are done:

[gravityvi]

Cucumber step definitions need to return a promise from the function. In this case, the step-definition should be like this:

Then(/^Search input should be visible$/,  function() {
    return browser.waitForElementVisible('[data-test-id="search-form-input"]');
       .verify.not.visible('[data-test-id="search-form-input"]');
});

[pawlakmaly]

When I modify the code as you said I get the same result as in my comment: You get the report before the assertion is finished...or even started.

I've checked that example from repo, seems to be working ok, but I have 0 idea why my code is not working

[gravityvi]

I think it is due to the behaviour of verify commands can you try using assert instead of verify?

[pawlakmaly]

@gravityvi Sorry for late response. I've changed 'verify' into 'assert' and it worked as expected. Seems that the verify api cause the issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any recent activity. If possible, please retry using the latest Nightwatch version and update the issue with any relevant details. If no further activity occurs, it will be closed. Thank you for your contribution.