Task does not fail on failed tests

[arendjr]

When a Casper test fails, I believe the Grunt task should fail and thus block further execution of other tasks and let Grunt return a non-zero status code.

Currently this is not the case, but I have been able to achieve this effect with a simple patch. In the file lib/casperjs.js I replaced the puts function with the following version:

function puts(error, stdout, stderr) {
  grunt.log.write('\nRunning tests from "' + filepath + '":\n');
  grunt.log.write(stdout);

  if ( error !== null ) {
    callback(error);
  } else {
    if (stdout.indexOf("FAIL") > -1) {
        callback("Tests failed");
    } else {
        callback();
    }
  }
}

I'm not sure if this is really the best way to go about this, but please consider including this or a similar patch.

[ronaldlokers]

What version of the casperjs task did you use? I tested it in the latest version (1.0.5) and it just stops executing the other grunt tasks when there is a negative result when all casperjs tests are done.

Or do you also want to stop the execution of the casperjs task after the first failed test?

[arendjr]

I'm using 1.0.5 indeed, with Grunt.js 0.4.0. casperjs --version yields 1.0.2.

When introducing a failed test the failure will be visible on the console, but Grunt.js finishes with Done, without errors. and returns 0.

[chrisocast]

I ran into the same issue when running this as part of a deployment process. Some tests failed, but since Grunt continued working, the deployment process wasn't aware.

Maybe there could be a config setting for grunt-casperjs where you can choose to stop grunt.js after a test fail, but by default it continues?

[abahdanovich]

I have the same problem. I use

├── grunt@0.4.1
├── grunt-casperjs@1.0.6

[bradoyler]

@chrisocast @ronaldlokers does it make sense to add option flag for "failTaskOnfailure"

[arendjr]

I think it does :)

[TechNickAI]

Hey guys, I'm happy to help package this up and submit a new version to npm, but I'm too busy to do the work myself. Anyone want to code this up and submit a pull request?

[oliveiraev]

I can try to pick up this task. But, first, let's clarify some parameters:

I've liked the flag approach, but I think that the natural behavior is, really, exiting upon test failures. On the other hand - if I just want a code coverage - I should explicity inform this.

My preference touches backward compatibility. If someone, using the so-early-version isn't aware about this change, some auto-deploy tools will become to fail.

Thought something about an strict flag. If true, continue only if all green. Otherwise ignore test failures.

Just waiting for a concern about the default value.

"casperjs": {
    "continueAlways": {
        "src": "this-test-has-failures.js",
        "strict": false
    },
    "continueOnlyIfAllTestsPasses": {
        "src": "this-test-should-pass.js",
        "strict": true
    }
}

[TechNickAI]

@oliveiraev - makes sense. Good balance between the intuitive behavior and backward compatibility.

[oliveiraev]

Hey, guys. Just remembered this issue and picked it up. Updated my master branch to reflect yours and, for my surprise, it acts like @ronaldlokers said at the first comment.

test.js:

casper.test.begin("Dummy failed test", function (tester) {
    tester.assert(false, "Hello from failed CasperJS test!");
    tester.done();
});

output

$ grunt casperjs:webkit           
Running "casperjs:webkit" (casperjs) task
Command: project/path/node_modules/grunt-casperjs/casperjs test test.js

Test file: project/path/test.js 
# Dummy failed test
FAIL Hello from failed CasperJS test!
#    type: assert
#    file: project/path/test.js:2
#    code: tester.assert(false, "Hello from failed CasperJS test!");
#    subject: false
FAIL 1 test executed in 0.037s, 0 passed, 1 failed, 0 dubious, 0 skipped.       

Details for the 1 failed test:

In project/path/test.js:2
  Dummy failed test
    assert: Hello from failed CasperJS test!                                    
Warning: Command failed:  Use --force to continue.

Aborted due to warnings.

[TechNickAI]

@jatin any comments?

[jatin]

@arendjr, @chrisocast looks like this was a bug in certain versions of casperjs

@gorillamania close this

[Pascalmh]

I'm having the same issue (Tests continue even though there was an error) with casperjs 1.1.0-beta3. Never the less I think it would be nice to have an Option to have one profile that will continue running on errors and one that will stop on errors (might come in handy with CI).