search

nightwatchjs / nightwatch

Integrated end-to-end testing framework written in Node.js and using W3C Webdriver API. Developed at @browserstack
https://nightwatchjs.org
MIT License
11.78k stars 1.31k forks source link

skip_testcases_on_fail: false not working in version 2.6.1 #3575

Closed ZScannell closed 1 year ago

ZScannell commented 1 year ago

Description of the bug/issue

Once a test fails, I'd like the test suites to continue running so that I can see all errors at once rather than having to sift through them one at a time.

I can do this by setting skip_testcases_on_fail: false inside the nightwatch.conf.js file.

This is on the latest version 2.6.1.

Steps to reproduce

  1. Run npm init nightwatch@latest (from https://nightwatchjs.org/guide/quickstarts/create-and-run-a-nightwatch-test.html)
  2. Add skip_testcases_on_fail: false to the test_settings json inside of nightwatch.conf.js.
  3. Create a reduced test case (attached as sample test) with a test intended to fail preceding a test intended to pass.
  4. Run npm nightwatch .\test\reducedTestCase.js.

Sample test

describe('Reduced Test Case - skipFailure not working', () => {
    before(browser => {
      browser.navigateTo('https://www.google.com/');
    });

    it('Title test case', function(browser) {
      browser.assert.titleEquals('Not Google');
      browser.assert.titleEquals('Google');
    });

    after(browser => {
      browser.end();
    });
  });

Command to run

npm nightwatch .\test\reducedTestCase.js

Verbose Output

[Reduced Test Case - skipFailure not working] Test Suite
───────────────────────────────────────────────────────────────────────────────
⠋ Starting ChromeDriver on port 9515...
⠙ Starting ChromeDriver on port 9515...
   Request POST /session  
   {
     capabilities: {
       firstMatch: [ {} ],
       alwaysMatch: { browserName: 'chrome', 'goog:chromeOptions': {} }
     }
⠧ Starting ChromeDriver on port 9515...

⠧ Starting ChromeDriver on port 9515...
   Response 200 POST /session (1524ms)
   {
     value: {
       capabilities: {
         acceptInsecureCerts: false,
         browserName: 'chrome',
         browserVersion: '109.0.5414.75',
         chrome: {
           chromedriverVersion: '109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414@{#1172})',
           userDataDir: 'C:\\Users\\ZSCANN~1\\AppData\\Local\\Temp\\1\\scoped_dir22220_1773750296'
         },
         'goog:chromeOptions': { debuggerAddress: 'localhost:53580' },
         networkConnectionEnabled: false,
         pageLoadStrategy: 'normal',
         platformName: 'windows',
         proxy: {},
         setWindowRect: true,
         strictFileInteractability: false,
         timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
         unhandledPromptBehavior: 'dismiss and notify',
         'webauthn:extension:credBlob': true,
         'webauthn:extension:largeBlob': true,
         'webauthn:virtualAuthenticators': true
       },
       sessionId: 'd036784128087cffb41fbbcdb99b15aa'
     }
ℹ Connected to ChromeDriver on port 9515 (1724ms).
  Using: chrome (109.0.5414.75) on WINDOWS.

 Received session with ID: d036784128087cffb41fbbcdb99b15aa

 → Running [before]:

 → Running command: navigateTo ('https://www.google.com/')
   Request POST /session/d036784128087cffb41fbbcdb99b15aa/url  
   { url: 'https://www.google.com/' }
   Response 200 POST /session/d036784128087cffb41fbbcdb99b15aa/url (1002ms)
   { value: null }
  → Completed command: navigateTo ('https://www.google.com/') (1007ms)
 → Completed [before].

  Running Title test case:
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 → Running [beforeEach]:
 → Completed [beforeEach].

 → Running command: assert.titleEquals ('Not Google')

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (53ms)
   { value: 'Google' }
  → Completed command: title ([Function]) (57ms)

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (32ms)
   { value: 'Google' }
  → Completed command: title ([Function]) (34ms)

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (16ms)
   { value: 'Google' }
  → Completed command: title ([Function]) (19ms)

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (13ms)
   { value: 'Google' }
  → Completed command: title ([Function]) (15ms)
[10196:22824:0118/101737.091:ERROR:device_event_log_impl.cc(215)] [10:17:37.091] USB: usb_device_handle_win.cc:1046 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[10196:22824:0118/101737.093:ERROR:device_event_log_impl.cc(215)] [10:17:37.093] USB: usb_device_handle_win.cc:1046 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[10196:22824:0118/101737.099:ERROR:device_event_log_impl.cc(215)] [10:17:37.098] USB: usb_device_handle_win.cc:1046 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[10196:22824:0118/101737.099:ERROR:device_event_log_impl.cc(215)] [10:17:37.099] USB: usb_device_handle_win.cc:1046 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[10196:22824:0118/101737.100:ERROR:device_event_log_impl.cc(215)] [10:17:37.099] USB: usb_device_handle_win.cc:1046 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (46ms)
   { value: 'Google' }
  → Completed command: title ([Function]) (58ms)
[3860:25772:0118/101737.400:ERROR:ssl_client_socket_impl.cc(985)] handshake failed; returned -1, SSL error code 1, net_error -101
[3860:25772:0118/101737.420:ERROR:ssl_client_socket_impl.cc(985)] handshake failed; returned -1, SSL error code 1, net_error -101

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (14ms)
   { value: 'Google' }
  → Completed command: title ([Function]) (17ms)

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (14ms)
   { value: 'Google' }
  → Completed command: title ([Function]) (17ms)

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (14ms)
   { value: 'Google' }
  → Completed command: title ([Function]) (17ms)

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (9ms)
   { value: 'Google' }
  → Completed command: title ([Function]) (15ms)

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (14ms)
   { value: 'Google' }
  → Completed command: title ([Function]) (18ms)

 → Running command: title ([Function])
   Request GET /session/d036784128087cffb41fbbcdb99b15aa/title  

   Response 200 GET /session/d036784128087cffb41fbbcdb99b15aa/title (8ms)
   { value: 'Google' }
  ✖ NightwatchAssertError
   Testing if the page title equals 'Not Google' in 5000ms - expected "is 'Not Google'" but got: "Google" (5508ms)

    Error location:
    C:\Users\zscannell\AutomatedTesting\test\reducedTestCase.js:
    –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     6 |
     7 |     it('Title test case', function(browser) {
     8 |       browser.assert.titleEquals('Not Google'); 
     9 |       browser.assert.titleEquals('Google');
     10 |     });
    –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

  → Completed command: title ([Function]) (26ms)
  → Completed command: assert.titleEquals ('Not Google') (5524ms)
 → Running [afterEach]:
 → Completed [afterEach].

  FAILED: 1 assertions failed (5.599s)
 → Running [after]:

 → Running command: end ()

 → Running command: session ('delete', [Function])
   Request DELETE /session/d036784128087cffb41fbbcdb99b15aa  

   Response 200 DELETE /session/d036784128087cffb41fbbcdb99b15aa (196ms)
   { value: null }
  → Completed command: end () (219ms)
  → Completed command: session ('delete', [Function]) (207ms)
 → Completed [after].

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  ️TEST FAILURE (11.716s):
   - 1 assertions failed; 0 passed

   × 1) test\reducedTestCase

   – Title test case (5.599s)

   → ✖ NightwatchAssertError
   Testing if the page title equals 'Not Google' in 5000ms - expected "is 'Not Google'" but got: "Google" (5508ms)

    Error location:
    C:\Users\zscannell\AutomatedTesting\test\reducedTestCase.js:
    –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     6 |
     7 |     it('Title test case', function(browser) {
     8 |       browser.assert.titleEquals('Not Google'); 
     9 |       browser.assert.titleEquals('Google');
     10 |     });
    –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

Nightwatch Configuration

module.exports = {
  // An array of folders (excluding subfolders) where your tests are located;
  // if this is not specified, the test source must be passed as the second argument to the test runner.
  src_folders: ['test','nightwatch/examples'],

  // See https://nightwatchjs.org/guide/concepts/page-object-model.html
  page_objects_path: ['nightwatch/page-objects'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-commands.html
  custom_commands_path: ['nightwatch/custom-commands'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-assertions.html
  custom_assertions_path: ['nightwatch/custom-assertions'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-plugins.html
  plugins: [],

  // See https://nightwatchjs.org/guide/concepts/test-globals.html
  globals_path: '',

  webdriver: {},

  test_workers: {
    enabled: true
  },

  test_settings: {
    default: {
      skip_testcases_on_fail: false,
      disable_error_log: false,
      launch_url: 'http://localhost',

      screenshots: {
        enabled: false,
        path: 'screens',
        on_failure: true
      },

      desiredCapabilities: {
        browserName: 'chrome'
      },

      webdriver: {
        start_process: true,
        server_path: ''
      },

    },

    firefox: {
      desiredCapabilities: {
        browserName: 'firefox',
        alwaysMatch: {
          acceptInsecureCerts: true,
          'moz:firefoxOptions': {
            args: [
              // '-headless',
              // '-verbose'
            ]
          }
        }
      },
      webdriver: {
        start_process: true,
        server_path: '',
        cli_args: [
          // very verbose geckodriver logs
          // '-vv'
        ]
      }
    },

    chrome: {
      desiredCapabilities: {
        browserName: 'chrome',
        'goog:chromeOptions': {
          // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/
          //
          // w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78)
          w3c: true,
          args: [
            //'--no-sandbox',
            //'--ignore-certificate-errors',
            //'--allow-insecure-localhost',
            //'--headless'
          ]
        }
      },

      webdriver: {
        start_process: true,
        server_path: '',
        cli_args: [
          // --verbose
        ]
      }
    },

  },

  usage_analytics: {
    enabled: true,
    log_path: './logs/analytics',
    client_id: '5aee7a6c-ec08-491b-a9c3-f2a5ef371d2a'
  }

};

Nightwatch.js Version

2.6.1

Node Version

18.13.0

Browser

Firefox 108.0.2; Chrome 109.0.5414.75

Operating System

Windows 10 Pro 19044.2364

Additional Information

No response

gravityvi commented 1 year ago

Hey @ZScannell, thanks for raising this issue. skip_testcases_on_fail determines whether to skip the test cases after the previous test case has failed. I think in this case to keep the commands running after a failure we can replace assert keyword with verify. Verify will still report errors but will not stop executing subsequent commands. Let me know if this works

ZScannell commented 1 year ago

Hello @gravityvi - I just re-ran the above sample test using verify instead of assert and can confirm that it is operating as expected now. I am closing the issue with the comment. Thank you for your help!