search

nightwatchjs / nightwatch

Integrated end-to-end testing framework written in Node.js and using W3C Webdriver API. Developed at @browserstack
https://nightwatchjs.org
MIT License
11.78k stars 1.31k forks source link

`end_session_on_fail` stopped working in version 2.0.4 #3574

Closed joshua-redmond closed 1 year ago

joshua-redmond commented 1 year ago

Description of the bug/issue

Once a test fails, I'd like to keep the browser open so I can see what went wrong.

I can do this by setting end_session_on_fail to false, which works perfectly on v1.7.13.

Once I upgraded to v2.6.8 (latest) or v2.0.4 (first 2.x version) it stopped working :thinking:

end_session_on_fail still exists in the documentation, so I guess it was just overlooked in the 2.x version?

Thanks.

Steps to reproduce

  1. Run: npm init nightwatch (from https://nightwatchjs.org/guide/quickstarts/create-and-run-a-nightwatch-test.html)
  2. Add end_session_on_fail: false, to: nightwatch.conf.js
  3. Put an x in the .titleContains at nightwatch/examples/basic/ecosia.js:7 (so the test fails).
  4. Run: npx nightwatch ./nightwatch/examples/basic/ecosia.js (from the same guide)

Sample test

// Copied from the example `ecosia.js` test.

describe('Ecosia.org Demo', function() {
  before(browser => browser.navigateTo('https://www.ecosia.org/'));

  it('Demo test ecosia.org', function(browser) {
    browser
      .waitForElementVisible('body')
      .assert.titleContains('Wrong title')
  });

  after(browser => browser.end());
});

Command to run

npx nightwatch ./nightwatch/examples/basic/ecosia.js

Verbose Output

[Ecosia.org Demo] Test Suite
────────────────────────────────────────────────────────
⠋ Starting ChromeDriver on port 9515...
 Starting ChromeDriver with server_path=/c/laptop/stuff/mike/junk/node_modules/chromedriver/lib/chromedriver/chromedriver...
   Request POST /session  
   {
     capabilities: {
       firstMatch: [ {} ],
       alwaysMatch: { browserName: 'chrome', 'goog:chromeOptions': {} }
     }
⠸ Starting ChromeDriver on port 9515...
   Response 200 POST /session (253ms)
   {
     value: {
       capabilities: {
         acceptInsecureCerts: false,
         browserName: 'chrome',
         browserVersion: '109.0.5414.74',
         chrome: {
           chromedriverVersion: '109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414@{#1172})',
           userDataDir: '/tmp/.com.google.Chrome.a5XRg4'
         },
         'goog:chromeOptions': { debuggerAddress: 'localhost:44841' },
         networkConnectionEnabled: false,
         pageLoadStrategy: 'normal',
         platformName: 'linux',
         proxy: {},
         setWindowRect: true,
         strictFileInteractability: false,
         timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
         unhandledPromptBehavior: 'dismiss and notify',
         'webauthn:extension:credBlob': true,
         'webauthn:extension:largeBlob': true,
         'webauthn:virtualAuthenticators': true
       },
       sessionId: '448ef5c5e3e8ce7338d108f6803d06c6'
     }
ℹ Connected to ChromeDriver on port 9515 (281ms).
  Using: chrome (109.0.5414.74) on LINUX.

 Received session with ID: 448ef5c5e3e8ce7338d108f6803d06c6

 → Running [before]:

 → Running command: navigateTo ('https://www.ecosia.org/')
   Request POST /session/448ef5c5e3e8ce7338d108f6803d06c6/url  
   { url: 'https://www.ecosia.org/' }
   Response 200 POST /session/448ef5c5e3e8ce7338d108f6803d06c6/url (2032ms)
   { value: null }
  → Completed command: navigateTo ('https://www.ecosia.org/') (2034ms)
 → Completed [before].

  Running Demo test ecosia.org:
──────────────────────────────────────────────────────────────────────────────────────────────────────
 → Running [beforeEach]:
 → Completed [beforeEach].

 → Running command: waitForElementVisible ('body')
   Request POST /session/448ef5c5e3e8ce7338d108f6803d06c6/elements  
   { using: 'css selector', value: 'body' }
   Response 200 POST /session/448ef5c5e3e8ce7338d108f6803d06c6/elements (9ms)
   {
     value: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'e778edd8-78e0-49c6-9c08-77854bae6fb8'
       }
     ]
  }
   Request POST /session/448ef5c5e3e8ce7338d108f6803d06c6/execute/sync  
   {
     script: 'return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.execScript("var "+a... (44027 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'e778edd8-78e0-49c6-9c08-77854bae6fb8',
         ELEMENT: 'e778edd8-78e0-49c6-9c08-77854bae6fb8'
       }
     ]
  }
   Response 200 POST /session/448ef5c5e3e8ce7338d108f6803d06c6/execute/sync (12ms)
   { value: true }
  ✔ Element <body> was visible after 23 milliseconds.
  → Completed command: waitForElementVisible ('body') (25ms)

 → Running command: assert.titleContains ('Wrong title')

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (4ms)
   { value: 'Ecosia - the search engine that plants trees' }
  → Completed command: title ([Function]) (6ms)

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (5ms)
   { value: 'Ecosia - the search engine that plants trees' }
  → Completed command: title ([Function]) (6ms)

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (7ms)
   { value: 'Ecosia - the search engine that plants trees' }
  → Completed command: title ([Function]) (8ms)

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (11ms)
   { value: 'Ecosia - the search engine that plants trees' }
  → Completed command: title ([Function]) (12ms)

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (17ms)
   { value: 'Ecosia - the search engine that plants trees' }
  → Completed command: title ([Function]) (21ms)

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (4ms)
   { value: 'Ecosia - the search engine that plants trees' }
  → Completed command: title ([Function]) (5ms)

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (13ms)
   { value: 'Ecosia - the search engine that plants trees' }
  → Completed command: title ([Function]) (15ms)

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (12ms)
   { value: 'Ecosia - the search engine that plants trees' }
  → Completed command: title ([Function]) (14ms)

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (13ms)
   { value: 'Ecosia - the search engine that plants trees' }
  → Completed command: title ([Function]) (15ms)

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (13ms)
   { value: 'Ecosia - the search engine that plants trees' }
  → Completed command: title ([Function]) (15ms)

 → Running command: title ([Function])
   Request GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title  

   Response 200 GET /session/448ef5c5e3e8ce7338d108f6803d06c6/title (6ms)
   { value: 'Ecosia - the search engine that plants trees' }
  ✖ NightwatchAssertError
   Testing if the page title contains 'Wrong title' in 5000ms - expected "contains 'Wrong title'" but got: "Ecosia - the search engine that plants trees" (5149ms)

    Error location:
    /c/laptop/stuff/mike/junk/nightwatch/examples/basic/ecosia.js:
    –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     5 |     browser
     6 |       .waitForElementVisible('body')
     7 |       .assert.titleContains('Wrong title') 
     8 |   });
     9 | 
    –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

  → Completed command: title ([Function]) (11ms)
  → Completed command: assert.titleContains ('Wrong title') (5155ms)
 → Running [afterEach]:
 → Completed [afterEach].

  FAILED: 1 assertions failed and  1 passed (5.234s)
 → Running [after]:

 → Running command: end ()

 → Running command: session ('delete', [Function])
   Request DELETE /session/448ef5c5e3e8ce7338d108f6803d06c6  

   Response 200 DELETE /session/448ef5c5e3e8ce7338d108f6803d06c6 (52ms)
   { value: null }
  → Completed command: session ('delete', [Function]) (54ms)
 Wrote log file to: /c/laptop/stuff/mike/junk/logs/basic/ecosia_chromedriver.log
  → Completed command: end () (59ms)
 → Completed [after].

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  ️TEST FAILURE (7.942s):  
   - 1 assertions failed; 1 passed

   ✖ 1) basic/ecosia

   – Demo test ecosia.org (5.234s)

   → ✖ NightwatchAssertError
   Testing if the page title contains 'Wrong title' in 5000ms - expected "contains 'Wrong title'" but got: "Ecosia - the search engine that plants trees" (5149ms)

    Error location:
    /c/laptop/stuff/mike/junk/nightwatch/examples/basic/ecosia.js:
    –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     5 |     browser
     6 |       .waitForElementVisible('body')
     7 |       .assert.titleContains('Wrong title') 
     8 |   });
     9 | 
    –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

 ChromeDriver process closed.

Nightwatch Configuration

// This whole thing was generated by: `npm init nightwatch`
// The only thing I changed was adding: `end_session_on_fail: false`
module.exports = {
  end_session_on_fail: false,
  // An array of folders (excluding subfolders) where your tests are located;
  // if this is not specified, the test source must be passed as the second argument to the test runner.
  src_folders: ['test','nightwatch/examples'],

  // See https://nightwatchjs.org/guide/concepts/page-object-model.html
  page_objects_path: ['nightwatch/page-objects'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-commands.html
  custom_commands_path: ['nightwatch/custom-commands'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-assertions.html
  custom_assertions_path: ['nightwatch/custom-assertions'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-plugins.html
  plugins: [],

  // See https://nightwatchjs.org/guide/concepts/test-globals.html
  globals_path: '',

  webdriver: {},

  test_workers: {
    enabled: true
  },

  test_settings: {
    default: {
      disable_error_log: false,
      launch_url: 'http://localhost',

      screenshots: {
        enabled: false,
        path: 'screens',
        on_failure: true
      },

      desiredCapabilities: {
        browserName: 'chrome'
      },

      webdriver: {
        start_process: true,
        server_path: ''
      },

    },

    chrome: {
      desiredCapabilities: {
        browserName: 'chrome',
        'goog:chromeOptions': {
          // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/
          //
          // w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78)
          w3c: true,
          args: [
            //'--no-sandbox',
            //'--ignore-certificate-errors',
            //'--allow-insecure-localhost',
            //'--headless'
          ]
        }
      },

      webdriver: {
        start_process: true,
        server_path: '',
        cli_args: [
          // --verbose
        ]
      }
    },

  },

};

Nightwatch.js Version

2.6.10

Node Version

16.19.0

Browser

Chrome 109.0.5414.74

Operating System

Ubuntu 20.04.5 LTS

Additional Information

Poking around the Nightwatch code a little, I found this: node_modules/nightwatch/lib/core/client.js:460

  /**
   * @deprecated
   */
  endSessionOnFail(val) {

It has @deprecated... so I guess this option is deprecated? The docs don't say it's deprecated though.

beatfactor commented 1 year ago

Thanks for raising this issue, we'll look into it. The @deprecated message there just means that the method endSessionOnFail is deprecated because it's not used anymore. But the setting itself should work as expected.