Exit code for node tests is 0 when tests fails

[c089]

Running this test results in exit code being set to 0.

[cjohansen]

Think I figured it out. I had the same problem in a project, and it happened because I used the globally installed buster binary while the project had buster also installed locally. Is that the case for you too?

[c089]

Nope, using node_modules/buster/bin/buster-test also exits with code 0 on failing node test.

[c089]

Actually, "node_modules/.bin/buster-test -e node" returns 1, only without the -e node it returns 0, so it might be related to #229.

[c089]

Seems to be a combination of both: buster-test: 0 node_modules/.bin/buster-test: 0 buster-test -e node: 0 node_modules/.bin/buster-test -e node: 1

[cjohansen]

Thanks, I'll investigate.

[cowwoc]

Please note that node <test> is also returning 0 on failed tests.

[dwittner]

@c089, could you please check, if the problem still exists (as you already did for #231?

[c089]

current status: both buster-test -e node and buster-test return 0 when running the gist I posted in the original ticket.

[dwittner]

Have you removed the statement var buster = require('buster'); from the configuration file before? Having it set i also got a return code 0.

[c089]

I now get 0 in any case: with and without the -e node, with and without the require('buster')

[dwittner]

I cant't reproduce that behaviour yet. For your test i get a return code 1 without the require('buster') statement in the configuration file, for buster-test and buster-test -e node, on a linux and windows system. If i add the require('buster') statement to the configuration file, i get always a return code 0.

buster.js: 0.6.12 Beta 4 node.js: Windows: v0.10.0, Ubuntu: v0.10.15

Could you please provide more details about your installation/configuration and how you execute your tests (manually or via shell script for example)?

[c089]

I used a locally installed buster, with no global one in the PATH (running node_modules/.bin/buster-test)

[dwittner]

@c089, What is the value of the NODE_PATH?

More and more i think @cjohansen was already on the right way. I now installed a local version of Buster.JS in addition to a global version. If i run the global version by buster-test i can see, that at some point a switch is made from the global to the local version.

runConfigGroups
runConfig
loadRunner
loadRunner.callback
analyzer.run
node-runner.run
testRun.start
testRun.runTests
autoRun
autoRun.run 2
testRunner.create 2
foo: F
Failure: foo bar
    [assert.isTrue] Expected false to be true
    at Object.buster.testCase.bar (./fail.js:6:16)

1 test case, 1 test, 1 assertion, 1 failure, 0 errors, 0 timeouts
Finished in 0.05s

As you can see the switch is made at autoRun.run (all added log messages for the local version end with "2"). That switching finally leads to an exit code 0.

@c089, I assume that in your case a switch from the local to the global version is made.

[c089]

I don't have a global version installed or any other version in the PATH. During the run, process.env.NODE_PATH is undefined.

[dwittner]

@c089, Then i think @cjohansen was totally on the wrong way ;)

Can you please execute the following commands in your project folder (manually, not from within a shell script) and post the result?:

node -v
node_modules/.bin/buster -v
echo $NODE_PATH
node_modules/.bin/buster-test
echo $?

What OS do you use, Linux or OS X?

[c089]

$ node -v
v0.10.15

$ node_modules/.bin/buster -v
Buster.JS version 0.6.12 Beta 4

$ echo $NODE_PATH

$ node_modules/.bin/buster-test
foo: F
Failure: foo bar
    [assert.isTrue] Expected false to be true
    at Object.buster.testCase.bar (./fail.js:6:16)

1 test case, 1 test, 1 assertion, 1 failure, 0 errors, 0 timeouts
Finished in 0.011s

$ echo $?
0

[dwittner]

@c089, Strange. It seems i have the same configuration than you, but i can't reproduce the issue.

buster@ubuntu:~/issues/227$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.2 LTS
Release:    12.04
Codename:   precise

buster@ubuntu:~/issues/227$ node -v
v0.10.15

buster@ubuntu:~/issues/227$ ./node_modules/.bin/buster -v
Buster.JS version 0.6.12 Beta 4

buster@ubuntu:~/issues/227$ tree -a -L 2
.
├── buster.js
├── fail.js
└── node_modules
    ├── .bin
    └── buster

3 directories, 2 files

buster@ubuntu:~/issues/227$ cat buster.js
//var buster = require('buster');
var config = module.exports;
config['node tests'] = {
    environment: 'node',
    tests: [ 'fail.js' ]
};

buster@ubuntu:~/issues/227$ cat fail.js 
var buster = require('buster');
var assert = buster.assertions.assert;

buster.testCase('foo', {
    'bar': function() {
        assert.isTrue(false);
    }
});

buster@ubuntu:~/issues/227$ echo $NODE_PATH

buster@ubuntu:~/issues/227$ ./node_modules/.bin/buster-test
foo: F
Failure: foo bar
    [assert.isTrue] Expected false to be true
    at Object.buster.testCase.bar (./fail.js:6:16)

1 test case, 1 test, 1 assertion, 1 failure, 0 errors, 0 timeouts
Finished in 0.081s

buster@ubuntu:~/issues/227$ echo $?
1

As mentioned earlier, i have to remove the statement require('buster') from the configuration file to get an exit code 1.

Can you reproduce the issue on other projects and on other machines as well?

[c089]

Doh, you're right - I tried again and it seems while testing the commands for my last comment, I cloned the gist again and forgot to remove the require statement again - I am now also getting the exit code 1. Sorry for that.

[dwittner]

Never mind! That can happen. I am glad, that the problem is solved.