Regression: Custom reporter not working for multiple test files execution

[mdharamadas]

This seems broken since v0.9.0 release. Custom reporter works with just one test file execution but doesn't work when executing a set of test files. This is for both approaches, --reporter and reporter method in external globals file.

I tried with v0.8.18 and it works fine for any test executions.

Its surprising that no one even reported this issue (does that mean nobody is using this feature? )

[bitwiseman]

@mdharamadas - How is the custom reporter broken? What is the command line or settings you're using? What happens in 0.8.18? What happens in 0.9.0?

[mdharamadas]

@bitwiseman pls see the required details below...

nightwatch.json

{
  "src_folders": ["tests"],
  "output_folder": "reports",
  "custom_commands_path": [],
  "custom_assertions_path": [],
  "page_objects_path": "lib/page/",
  "globals_path": "./globalsModule.js",
  "live_output": false,

  "test_workers": {
    "enabled": true,
    "workers": "auto"
  },

  "test_settings": {
    "default": {
      "skip_testcases_on_fail": false,
      "end_session_on_fail": true,
      "silent": true,
      "screenshots": {
        "enabled": true,
        "on_failure": true,
        "path": "screenshots"
      },
      "selenium": {
        "log_path": "log/"
      },
      "desiredCapabilities": {
        "browserName": "chrome",
        "javascriptEnabled": true,
        "acceptSslCerts": true
      }
    },
    "chrome": {
      "launch_url": "http://localhost",
      "selenium_port": 4444,
      "selenium_host": "localhost",
      "selenium": {
        "start_process": true,
        "server_path": "",
        "host": "127.0.0.1",
        "port": 4444,
        "cli_args": {
          "webdriver.chrome.driver": ""
        }
      }
    }
  }
}

In globalsModule.js, I have reporter funtion defined as follows...

...
  reporter: function (results, done) {
    console.log('custom reporting');
    done();
  }
...

When I run single test file using following command line, I see that it prints 'custom reporting' in the test execution logs. ./nightwatch --config ./nightwatch.conf.js -e chrome --test tests/sample.js

But, when I run all the test files using following command line, I DO NOT see 'custom reporting' printed in the test execution logs. ./nightwatch --config ./nightwatch.conf.js -e chrome

Both of these commands work as expected (prints 'custom reporting' in logs) with Nightwatch v0.8.18.

Same behaviour if I have separate reporter module defined in a separate file and call it using command line option --reporter.

Let me know if you need any further details or if anythings not clear.

[mdharamadas]

@beatfactor can you take a look at this one pls? thanks!

[Bobu77]

Same issue for me, that'd help a lot! Thanks!

[termleech]

I can confirm this is still an issue on 0.9.16

[xeppen]

+1

[StephanBijzitter]

Was browsing through issues to find another issue and stumbled upon this.

Guess mine is a duplicate, but I turned it into a feature request instead of a bug report as I was not aware this is a regression.

https://github.com/nightwatchjs/nightwatch/issues/1557

[nknapp]

We have digged into the code a little and found out the following:

• The Reporter-class is instantiated in two places: lib/runner/run.js and lib/runner/cli/clirunner.js
• The "options"-parameter is different in the two places and we assume that the lib/runner/cli/clirunner.js is called when running tests in a test-worker.
• Our custom reporter uses data from the globals.test_settings, which are only part of the options if the reporter is not called from a test-worker. This leads to an exception and the report-file is not written.

The correct solution in my opinion would be, not to call the reporter at all from the test-workers. Instead the worker should report back the results to the main-process. The main-process should merge the results and then call the reporter.

[Rajesh-Narayanappa87]

+1. Able to reproduce in 0.9.19

[ankiit]

I am able to write to a file from custom reporter but console.log doesn't work if custom reporter is defined in globalsModule.js. nightwatch version 0.9.20

[aberonni]

Hi @beatfactor can you give an update on this?

[beatfactor]

@aberonni have you tested it in the latest v1.0.6?

[aberonni]

@beatfactor reproduced also with v1.0.6

You can see the output with only 1 test result instead of 2 here, and reproduce in this repo.

[aberonni]

@beatfactor could you add the v1.0 tag to this issue?

Also let me know when the latest code is pushed and I will test again against the latest code.

[lreading]

Has anyone found a workaround for the interim?

[aberonni]

@lreading we have a workaround in place that uses the global afterEach hook. The client parameter contains a currentTest property with which we update a json.

This is a rough example (might need some readaption to work correctly).

const filePath = 'some/path/to.json';

afterEach: (client, done) => {
    const { currentTest } = client;
    let results;

    try {
        results = require(filePath);
    } catch (e) {
        results = {};
    }

    const data = {
        name: currentTest.module,
        success: currentTest.results.errors === 0 && currentTest.results.failed === 0,
        errors: Object.entries(currentTest.results.testcases).reduce((errors, [testcaseName, testcase]) => {
            if (testcase.errors !== 0 || testcase.failed !== 0) {
                errors.push({
                    testcase: testcaseName,
                    assertions: testcase.assertions
                });
            }

            return errors;
        }, [])
    };

    results[data.name] = data;

    try {
        fs.writeFileSync(filePath, JSON.stringify(results), "utf8");
    } catch (e) {
        console.log(`An error occured while saving ${filePath}:`);
        console.log(e);
    }

    done()
};

[lreading]

Thanks for the detailed response!

[stale[bot]]

This issue has been automatically marked as stale because it has not had any recent activity. If possible, please retry using the latest Nightwatch version and update the issue with any relevant details. If no further activity occurs, it will be closed. Thank you for your contribution.

[aberonni]

Not stale

[aberonni]

@beatfactor looks like this can be closed now that the PR has been merged and released

[mrkmhny]

Still experiencing this on v1.3.2. The results object that is passed into the write() method inside of the file specified with the --reporter flag is empty when running more than 1 test file.