Mocha returns exit status 0 on failure

[just-paja]

Just changed one of your tests to fail and discovered that Mocha will return exit status 0 when it fails. Should be 1.

mocha-multi-reporters-demo $ npm test

> mocha-multi-reporters-demo@1.0.12 test /home/just-paja/Work/git/fork/mocha-multi-reporters-demo
> mocha --reporter mocha-multi-reporters --reporter-options configFile=config.json

  mocha-test #1
    ✓ sample test #1.1
    1) sample test #1.2

  mocha-test #2
    ✓ sample test #2.1
    - sample test #2.2

  2 passing (14ms)
  1 pending
  1 failing

  1) mocha-test #1 sample test #1.2:
     AssertionError: expected true to be false
      at Context.<anonymous> (test/demo.js:15:27)

$ echo $?
0

When using mocha without this reporter

mocha-multi-reporters-demo $ npm test

> mocha-multi-reporters-demo@1.0.12 test /home/just-paja/Work/git/fork/mocha-multi-reporters-demo
> mocha

  mocha-test #1
    ✓ sample test #1.1
    1) sample test #1.2

  mocha-test #2
    ✓ sample test #2.1
    - sample test #2.2

  2 passing (14ms)
  1 pending
  1 failing

  1) mocha-test #1 sample test #1.2:
     AssertionError: expected true to be false
      at Context.<anonymous> (test/demo.js:15:27)

npm ERR! Test failed.  See above for more details.
mocha-multi-reporters-demo $ echo $?
1

[stanleyhlng]

Hi @just-paja,

I cannot report the error. May I have your current dev. environment settings?

This is my env:

➜  mocha-multi-reporters-demo git:(master) node --version
v6.5.0
➜  mocha-multi-reporters-demo git:(master) npm --version
3.10.7
➜  mocha-multi-reporters-demo git:(master) npm ls --depth 0
mocha-multi-reporters-demo@1.0.12 /Users/sng/Projects/stanleyhlng/github/mocha-multi-reporters-demo
├── chai@3.5.0
├── eslint@3.9.0
├── eslint-config-defaults@9.0.0
├── mocha@3.1.2
├── mocha-junit-reporter@1.12.1
├── mocha-multi-reporters@1.1.0
└── pre-commit@1.1.3

Execute Mocha test w/ mocha-multi-reporters

➜  mocha-multi-reporters-demo git:(master) ./node_modules/.bin/mocha  --reporter mocha-multi-reporters --reporter-options configFile=config.json
1..4
ok 1 mocha-test 1 sample test 1.1
ok 2 mocha-test 1 sample test 1.2
ok 3 mocha-test 2 sample test 2.1
ok 4 mocha-test 2 sample test 2.2 # SKIP -
# tests 3
# pass 3
# fail 0

➜  mocha-multi-reporters-demo git:(master) cat config.json
{
    "reporterEnabled": "tap,xunit,mocha-junit-reporter",
    "xunitReporterOptions": {
        "output": "xunit-custom.xml"
    },
    "mochaJunitReporterReporterOptions": {
    "mochaFile": "junit-custom.xml"
    }
}

➜  mocha-multi-reporters-demo git:(master) cat xunit-custom.xml
<testsuite name="Mocha Tests" tests="4" failures="0" errors="0" skipped="1" timestamp="Thu, 03 Nov 2016 22:27:34 GMT" time="0.007">
<testcase classname="mocha-test #1" name="sample test #1.1" time="0.001"/>
<testcase classname="mocha-test #1" name="sample test #1.2" time="0"/>
<testcase classname="mocha-test #2" name="sample test #2.1" time="0.001"/>
<testcase classname="mocha-test #2" name="sample test #2.2" time="0"><skipped/></testcase>
</testsuite>

➜  mocha-multi-reporters-demo git:(master) cat junit-custom.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0.002" tests="4" failures="0" skipped="1">
  <testsuite name="Root Suite" timestamp="2016-11-03T22:27:34" tests="0" failures="0" time="0">
  </testsuite>
  <testsuite name="mocha-test #1" timestamp="2016-11-03T22:27:34" tests="2" failures="0" time="0.001">
    <testcase name="mocha-test #1 sample test #1.1" time="0.001" classname="sample test #1.1">
    </testcase>
    <testcase name="mocha-test #1 sample test #1.2" time="0" classname="sample test #1.2">
    </testcase>
  </testsuite>
  <testsuite name="mocha-test #2" timestamp="2016-11-03T22:27:34" tests="2" failures="0" time="0.001">
    <testcase name="mocha-test #2 sample test #2.1" time="0.001" classname="sample test #2.1">
    </testcase>
  </testsuite>
</testsuites>%

➜  mocha-multi-reporters-demo git:(master) echo $!
0

Execute Mocha test w/o mocha-multi-reporters

➜  mocha-multi-reporters-demo git:(master) ./node_modules/.bin/mocha

  mocha-test #1
    ✓ sample test #1.1
    ✓ sample test #1.2

  mocha-test #2
    ✓ sample test #2.1
    - sample test #2.2

  3 passing (8ms)
  1 pending

➜  mocha-multi-reporters-demo git:(master) echo $!
0

[just-paja]

Hi, @stanleyhlng,

you're not checking exit status ($?), but last command process PID ($!).

[stanleyhlng]

Hi, @just-paja

I re-run the steps and check exist status ($?), it both return 0.

Execute Mocha test w/ mocha-multi-reporters

➜  mocha-multi-reporters-demo git:(master) ./node_modules/.bin/mocha --reporter mocha-multi-reporters --reporter-options configFile=config.json
1..4
ok 1 mocha-test 1 sample test 1.1
ok 2 mocha-test 1 sample test 1.2
ok 3 mocha-test 2 sample test 2.1
ok 4 mocha-test 2 sample test 2.2 # SKIP -
# tests 3
# pass 3
# fail 0

➜  mocha-multi-reporters-demo git:(master) echo $?
0

Execute Mocha test w/o mocha-multi-reporters

➜  mocha-multi-reporters-demo git:(master) ./node_modules/.bin/mocha

  mocha-test #1
    ✓ sample test #1.1
    ✓ sample test #1.2

  mocha-test #2
    ✓ sample test #2.1
    - sample test #2.2

  3 passing (8ms)
  1 pending

➜  mocha-multi-reporters-demo git:(master) echo $?
0

After checking your console result, it should pass the following test and expect no npm ERR! Test failed error. https://github.com/stanleyhlng/mocha-multi-reporters-demo/blob/master/test/demo.js#L14-L17

[stanleyhlng]

@just-paja I search around mocha repo and find this https://github.com/mochajs/mocha/issues/2177#issuecomment-210191212

[stanleyhlng]

@just-paja new package is released to fix the exit code issue.

https://github.com/stanleyhlng/mocha-multi-reporters/releases/tag/v1.1.2

[stanleyhlng]

please feel free reopen if you have any issue.