nightwatchjs / nightwatch

Integrated end-to-end testing framework written in Node.js and using W3C Webdriver API. Developed at @browserstack
https://nightwatchjs.org
MIT License
11.85k stars 1.35k forks source link

Screenshots are not taken on failure if mixing async test with sync afterEach test hook #3756

Open reallymello opened 1 year ago

reallymello commented 1 year ago

Description of the bug/issue

When I run my test with screenshots enabled on failure and my test fails I expect a screenshot to be saved under the screenshots directory of the current browser screen, but this is not happening when the test is async and I am using a non-async afterEach hook.

Steps to reproduce

  1. Create an async test that will fail
  2. Add an afterEach hook that is not async
  3. Run the test. No screenshot is saved.

Sample test

import { NightwatchBrowser } from 'nightwatch';

module.exports = {
  'should take a screenshot': async (browser: NightwatchBrowser) => {
    browser.assert.fail('oops');
  },

  afterEach: (browser: NightwatchBrowser) => {
    browser.navigateTo('https://www.ecosia.org/');
  },
};

Command to run

npx nightwatch .\nightwatch

Verbose Output

npx nightwatch .\nightwatch --verbose
 Now you can run TS tests directly using Nightwatch.
 Launching up to 1 concurrent test worker processes...

 Running:  default: test2.ts 

DevTools listening on ws://127.0.0.1:56766/devtools/browser/9448f3e0-9483-4ad3-9f50-69ddb5c04fc0

×  default: test2.ts   Now you can run TS tests directly using Nightwatch. 

 [Nightwatch\test2] Test Suite 
 ────────────────────────────────────────────────────────── 
 Starting ChromeDriver with server_path=C:\Users\Mr\Desktop\screenshotTest\node_modules\chromedriver\lib\chromedriver\chromedriver.exe... 
   Request POST /session   
 { 
      capabilities: { 
        firstMatch: [ {} ], 
        alwaysMatch: { browserName: 'chrome', 'goog:chromeOptions': {} } 
      } 
   } 
   Response 200 POST /session (633ms) 
 { 
      value: { 
        capabilities: { 
          acceptInsecureCerts: false, 
          browserName: 'chrome', 
          browserVersion: '113.0.5672.127', 
          chrome: { 
            chromedriverVersion: '113.0.5672.63 (0e1a4471d5ae5bf128b1bd8f4d627c8cbd55f70c-refs/branch-heads/5672@{#912})', 
            userDataDir: 'C:\\Users\\Mr\\AppData\\Local\\Temp\\scoped_dir13232_1753912520' 
          }, 
          'goog:chromeOptions': { debuggerAddress: 'localhost:56766' }, 
          networkConnectionEnabled: false,
          pageLoadStrategy: 'normal',
          platformName: 'windows',
          proxy: {},
          setWindowRect: true,
          strictFileInteractability: false,
          timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
          unhandledPromptBehavior: 'dismiss and notify',
          'webauthn:extension:credBlob': true,
          'webauthn:extension:largeBlob': true,
          'webauthn:extension:minPinLength': true,
          'webauthn:extension:prf': true,
          'webauthn:virtualAuthenticators': true
        },
        sessionId: 'e7922c34d80c125a32fdca1e4207430c'
      }
   }
 Using: chrome (113.0.5672.127) on WINDOWS.
 Received session with ID: e7922c34d80c125a32fdca1e4207430c 

 → Running [before]:
 → Completed [before].

 – should take a screenshot
 → Running [beforeEach]:
 → Completed [beforeEach].
 → Running [afterEach]:

  → Running command: assert.fail () 
  ✖ NightwatchAssertError
    oops

     Error location: 
     C:\Users\Mr\Desktop\screenshotTest\nightwatch\test2.ts:
     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––
      3 | module.exports = {
      4 |   'should take a screenshot': async (browser: NightwatchBrowser) => {
      5 |     browser.assert.fail('oops'); 
      6 |   },
      7 |
     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––

  → Completed command: assert.fail () (4ms)
 → Completed [afterEach].
 × default: test2.ts [Nightwatch\test2] should take a screenshot (37ms)
    oops
        at Object.should take a screenshot (C:\Users\Mr\Desktop\screenshotTest\nightwatch\test2.ts:5:20)
 → Running [after]:
 → Completed [after].

  → Running command: end (true) 

  → Running command: session ('delete', [Function]) 
   Request DELETE /session/e7922c34d80c125a32fdca1e4207430c  

   Response 200 DELETE /session/e7922c34d80c125a32fdca1e4207430c (88ms)
 { value: null }
  → Completed command: end (true) (108ms)
  → Completed command: session ('delete', [Function]) (92ms)

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  ️TEST FAILURE (3.425s):
   - 1 assertions failed; 0 passed

   × 1) nightwatch\test2

   – should take a screenshot (37ms)

   → ✖ NightwatchAssertError
   oops

    Error location:
    C:\Users\Mr\Desktop\screenshotTest\nightwatch\test2.ts:
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     3 | module.exports = {
     4 |   'should take a screenshot': async (browser: NightwatchBrowser) => {
     5 |     browser.assert.fail('oops'); 
     6 |   },
     7 |
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––

 Wrote HTML report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\nightwatch-html-report\index.html

 Wrote JSON report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\nightwatch\CHROME_113.0.5672.127__test2.json
 Wrote Rerun Json report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\minimal_report.json
 Wrote XML report file to: C:\Users\Mr\Desktop\screenshotTest\tests_output\nightwatch\CHROME_113.0.5672.127__test2.xml

Nightwatch Configuration

// Refer to the online docs for more details:
// https://nightwatchjs.org/gettingstarted/configuration/
//

//  _   _  _         _      _                     _          _
// | \ | |(_)       | |    | |                   | |        | |
// |  \| | _   __ _ | |__  | |_ __      __  __ _ | |_   ___ | |__
// | . ` || | / _` || '_ \ | __|\ \ /\ / / / _` || __| / __|| '_ \
// | |\  || || (_| || | | || |_  \ V  V / | (_| || |_ | (__ | | | |
// \_| \_/|_| \__, ||_| |_| \__|  \_/\_/   \__,_| \__| \___||_| |_|
//             __/ |
//            |___/

module.exports = {
  // An array of folders (excluding subfolders) where your tests are located;
  // if this is not specified, the test source must be passed as the second argument to the test runner.
  src_folders: ['test', 'nightwatch'],

  // See https://nightwatchjs.org/guide/concepts/page-object-model.html
  page_objects_path: [],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-commands.html
  custom_commands_path: [],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-assertions.html
  custom_assertions_path: [],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-plugins.html
  plugins: [],

  // See https://nightwatchjs.org/guide/concepts/test-globals.html
  globals_path: '',

  webdriver: {},

  test_workers: {
    enabled: true,
  },

  test_settings: {
    default: {
      disable_error_log: false,
      launch_url: 'http://localhost',

      screenshots: {
        enabled: true,
        path: 'screens',
        on_failure: true,
      },

      desiredCapabilities: {
        browserName: 'chrome',
      },

      webdriver: {
        start_process: true,
        server_path: '',
      },
    },

    chrome: {
      desiredCapabilities: {
        browserName: 'chrome',
        'goog:chromeOptions': {
          // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/
          //
          // w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78)
          w3c: true,
          args: [
            //'--no-sandbox',
            //'--ignore-certificate-errors',
            //'--allow-insecure-localhost',
            //'--headless'
          ],
        },
      },

      webdriver: {
        start_process: true,
        server_path: '',
        cli_args: [
          // --verbose
        ],
      },
    },
  },
};

Nightwatch.js Version

2.6.21

Node Version

18.16.0

Browser

Chrome 113

Operating System

Windows 10

Additional Information

If the afterEach test hook is marked async the screenshot is saved, but it seems like it should work in either case

❌ afterEach: (browser: NightwatchBrowser) => {

✅ afterEach: async (browser: NightwatchBrowser) => {

AutomatedTester commented 1 year ago

In your sample test you don't have an await. Could you retry the following snippet. Your async doesn't have an await.

import { NightwatchBrowser } from 'nightwatch';

module.exports = {
  'should take a screenshot': async (browser: NightwatchBrowser) => {
    await browser.assert.fail('oops');
  },

  afterEach: (browser: NightwatchBrowser) => {
    browser.navigateTo('https://www.ecosia.org/');
  },
};
reallymello commented 1 year ago

So, maybe a silly question, but if the test is marked async does it require an await? browser.assert doesn't typically need to be awaited so now I'm confused.

beatfactor commented 1 year ago

Preferably, yes. We can't detect if you're using await so the test will continue without any warning and in most cases I think it will be fine, but in some situations, like yours the next testcase or the afterEach hook will start sooner than expected, so the queue might get confused.

Even though in theory the internal queue should be able to handle these situations, it's best to try and avoid them.

reallymello commented 1 year ago

So in async test cases is the recommendation to await every line or ensure at least one await is used?

beatfactor commented 1 year ago

Yes, await every line, but you can still use chaining, you don't have to await every single command, you can await a chain as well.

simonwang384 commented 8 months ago

I'm running into a similar issue on nightwatch 3.5.0 where screenshots aren't being taken on failure.

Sample test:

import { NightwatchAPI } from 'nightwatch'
import { Google } from '../../page-objects'

describe('Google', function () {
  this.tags = ['google', 'ui']
  let google: Google

  before(function (browser: NightwatchAPI) {
    google = browser.page.google()
  })

  beforeEach(function() {
    google.navigate()
  })

  afterEach(function (browser: NightwatchAPI) {
    browser.end()
  })

  it('image should be visible', async function () {
    await browser.assert.fail('oops')
  })
})

I tried making the afterEach async and await browser.end() and that didn't work. Tried using the done callback and that didn't work either. Tried using arrow functions but that didn't work.

The only thing that did work was removing async from the test (it) but I would expect it to work with a async test function as well. It seems the like the issue is around having the test be marked as async.

Any ideas what could be happening or any recommendations? I can open a new issue if needed. I would prefer using async/await so I don't do too many nested call backs.