search

iamchrismiller / grunt-casper

Run CasperJS Scripts/Functional Tests
Other
69 stars 38 forks source link

Grunt should fail when casperjs fails #30

Closed schickling closed 10 years ago

schickling commented 10 years ago

screenshot 2014-02-22 22 44 26

iamchrismiller commented 10 years ago

Hmm.. I have a test that shows a different result. grunt casper:fail aborts with warnings.

What version of casperjs and this plugin are you using?

schickling commented 10 years ago

1.1.0-beta3

iamchrismiller commented 10 years ago

Does this still happen with 0.3.0? If so can you give me a bit more context?

What is your casperjs task config look like? How are you encountering this error?

schickling commented 10 years ago

https://travis-ci.org/schickling/calculate-size/builds/19435388 still fails with 0.3.0

bboyle commented 10 years ago

I just faced a similar issue, on Windows 8. I switched to grunt-casper v0.2.2 and grunt now fails when the test fails.

zikifer commented 10 years ago

Seeing the same issue (Windows 8.1) with version 0.3.10 and CasperJS 1.1.0-beta3. I downloaded the source and tried the failure unit test and it has the same problem, namely Grunt is reporting the tests ran without error.

C:\Users\sk\Downloads\grunt-casper-master>grunt casper:fail
Running "casper:fail" (casper) task
Test file: C:\Users\sk\Downloads\grunt-casper-master\test\fixtures\testFail.js
# Basic Site Testing Fail Tests
PASS Page title is: "Test Title"
PASS Header Exists
PASS P Tag Exists
FAIL Should Fail - Span Tag Does Not Exist
#    type: assertExists
#    file: C:\Users\sk\Downloads\grunt-casper-master\test\fixtures\testFail.js:8
#    code: test.assertExists('span', 'Should Fail - Span Tag Does Not Exist');
#    subject: false
#    selector: "span"
FAIL 4 tests executed in 0.097s, 3 passed, 1 failed, 0 dubious, 0 skipped.

Details for the 1 failed test:

In C:\Users\sk\Downloads\grunt-casper-master\test\fixtures\testFail.js:8
  Basic Site Testing Fail Tests
    assertExists: Should Fail - Span Tag Does Not Exist
Result log stored in tmp/casper/testFail-results.xml
Casper Task 'casper:fail' took ~2682ms to run

Done, without errors.

It's the Done, without errors that's the issue. Grunt is returning success when it should be returning failure, so the testing harness doesn't know something failed. In my case I'm kicking things off using ANT (from Maven via the maven-antrun-plugin) and looking at ${cmdresult} - it should contain a failure code but instead is 0 (success).

zikifer commented 10 years ago

I tracked it down - it's a bug in CasperJS 1.1-beta3. The good news is the bug is fixed in the HEAD revision on GIT, so the next version of CasperJS will have it fixed (be that 1.1-beta4 or 1.1 (final)). Who knows when that might be though...

iamchrismiller commented 10 years ago

I have fixed this issue in 0.4.0. Please let me know if it resolves your issues, @schickling @zikifer @bboyle

icd2k3 commented 9 years ago

Hey Chris, I still see the issue using SlimerJS as my engine in grunt-casper 0.4.1 with CasperJS 1.1-beta3 - details here: https://github.com/iamchrismiller/grunt-casper/issues/58

xzer commented 9 years ago

I got this issue for grunt-casper 0.4.2 and phantomjs 1.9.8 with casperjs 1.1-beta3