Test runner doesn't return correct failure code if run in node script

[gabaum10]

Take this setup for example:

• Node script that looks like this:

  "scripts": {
  "prequnit": "webpack --config ./webpack.config-qunit.js",
  "qunit": "node ./scripts/runQunit.js",
  }

The pre step compiles the necessary entry files for the tests. Then the runQunit.js file looks like this:

const qunit = require('node-qunit-phantomjs'); // eslint-disable-line

qunit('./src/test/resources/qunit/test1.js.html', {verbose: true});
qunit('./src/test/resources/qunit/test2.js.html', {verbose: true});
qunit('./src/test/resources/qunit/test3.js.html', {verbose: true});

If one of those tests fail, it prints an error to the console, but doesn't return a valid error code that would kill something like CircleCI. Is there a flag in the nodeJS implementation to cause the script to fail completely if any of the qunit calls fail?

[gabaum10]

I was able to work around it by adding

process.on('exit', () => {
    throw new Error('Unit tests failed.  See above for more information.');
});

to the top of my node script. Ugly, but I guess it works.

[gabaum10]

Actually, that didn't work. That returns an error code no matter what the exit code is. The problem is, the node execution of the qunit command always returns 0, for some reason. Even when it's failing.

[gabaum10]

I got it working, but it's really ugly and painful. I had to call the command line directly using node execSync... yeah.

const execSync = require('child_process').execSync;

execSync('node-qunit-phantomjs ./src/test/resources/qunit/test1.js.html --verbose', {stdio: [0, 1, 2]});
execSync('node-qunit-phantomjs ./src/test/resources/qunit/test12js.html --verbose', {stdio: [0, 1, 2]});

[jonkemp]

I don't know enough about this to propose a solution, so I'm open to suggestions or pull requests even.

[gabaum10]

Sure, I'll try and look into it further when I get a sec. For the time being, executing the commands directly via an execSync() works. Honestly, it might be just a short fall of node itself.