search

iamchrismiller / grunt-casper

Run CasperJS Scripts/Functional Tests
Other
69 stars 38 forks source link

Should fail when tests fail (v0.2.0 vs. v0.1.x) #23

Closed andacata closed 10 years ago

andacata commented 10 years ago

I'm executing my tests with the version 0.2.0 and the result is:

[warning] [phantom] Loading resource failed with status=fail: https://www.example.com/
[warning] [phantom] Casper.waitFor() timeout
FAIL Waiting for login form
#    type: fail
#    file: test/casperjs/tests/wallet.js
#    subject: false
[warning] [phantom] --fail-fast: aborted all remaining tests
⚠  --fail-fast: aborted all remaining tests
# Multiple bonus exchange.
FAIL 7 tests executed in 52.783s, 6 passed, 1 failed, 0 dubious, 0 skipped.

Details for the 1 failed test:

In test/casperjs/tests/wallet.js
  Canceled bonus buying.
    fail: Waiting for login form
>>
Casper Task 'casper:wallet' took ~53474ms to complete

Done, without errors.

$ echo $?
0

Executing the same tests with 0.1.4 the result was:

[warning] [phantom] Loading resource failed with status=fail: https://www.example.com/
[warning] [phantom] Casper.waitFor() timeout
FAIL Waiting for login form
#    type: fail
#    file: test/casperjs/tests/wallet.js
#    subject: false
[warning] [phantom] --fail-fast: aborted all remaining tests
⚠  --fail-fast: aborted all remaining tests
# Multiple bonus exchange.
FAIL 7 tests executed in 42.106s, 6 passed, 1 failed, 0 dubious, 0 skipped.

Details for the 1 failed test:

In test/casperjs/tests/wallet.js
  Canceled bonus buying.
    fail: Waiting for login form
>>
Warning: Task "casper:wallet" failed. Use --force to continue.

Aborted due to warnings.

$ echo $?
3
iamchrismiller commented 10 years ago

Looks like I had a slight issue with the Exit Logic. Fixed and published 0.2.1 to npm. Thank you for catching that.

rictorres commented 10 years ago

I'm still getting this on 0.2.1

Details for the 1 failed test:

In test/casperjs/test.js:9
  Open left sidebar
    assertUrlMatch: Current url matches the provided pattern
Result log stored in xunit/casper-results.xml
Casper Task 'casper:functional' took ~4562ms to run

Done, without errors.
iamchrismiller commented 10 years ago

Can you provide your config block please?

rictorres commented 10 years ago

Sure!

casper: {
  functional: {
    options: {
      test: true
    },
    files : {
      'xunit/casper-results.xml' : ['test/casperjs/test.js']
    }
  }
}
iamchrismiller commented 10 years ago

hmm, thats odd... I have a test for that in the test suite. Running The test from the cloned/node_module folder

$ grunt casper:fail

Running "casper:fail" (casper) task

Test file: test/fixtures/testFail.js                                            
# Basic Site Testing Fail Tests
PASS Page title is: "Test Title"
PASS Header Exists
PASS P Tag Exists
FAIL Should Fail - Span Tag Does Not Exist
#    type: assertExists
#    file: test/fixtures/testFail.js:8
#    code: test.assertExists('span', 'Should Fail - Span Tag Does Not Exist');
#    subject: false
#    selector: "span"
FAIL 4 tests executed in 0.069s, 3 passed, 1 failed, 0 dubious, 0 skipped.      
Details for the 1 failed test:

In test/fixtures/testFail.js:8
  Basic Site Testing Fail Tests
    assertExists: Should Fail - Span Tag Does Not Exist
Result log stored in tmp/casper/testFail-results.xml                            
>> 
Casper Task 'casper:fail' took ~700ms to run
Warning: Task "casper:fail" failed. Use --force to continue.

Aborted due to warnings.
rictorres commented 10 years ago

That's weird :(

I'm running

$ node -v
v0.10.24

$ grunt --version
grunt-cli v0.1.13
grunt v0.4.2

on Windows 7.

iamchrismiller commented 10 years ago

Okay. I will check on a virtual instance of Win7.

iamchrismiller commented 10 years ago

It will be a bit until I get an instance up and running.. I will have to look into it after work. My windows navigation is a bit rusty. Feel free the check it out. I am assuming it has something to do with the exit code as it works in OSX and Ubuntu.

andacata commented 10 years ago

Tested version 0.2.1 on Windows 7 with Cygwin, and running fine. Thanks.

iamchrismiller commented 10 years ago

Great. Thanks @andacata .. @rictorres does the issue still exist for you? I wasn't able to replicate it on a Virtualized instance of Windows7

rictorres commented 10 years ago 
Details for the 2 failed tests:

In test/casperjs/test.js:1355
  Open left sidebar
    uncaughtError: Cannot dispatch mousedown event on nonexistent selector: .exp
In test/casperjs/test.js:9
  Open left sidebar
    assertUrlMatch: Current url matches the provided pattern
Result log stored in xunit/casper-results.xml
Casper Task 'casper:functional' took ~5520ms to run

Done, without errors.

Still the same. Perhaps is something related to my setup. I'm going to get a new machine in a couple weeks, so I'm not going to worry about this right now :)

Thanks for taking the time to test it (again). @andacata @iamchrismiller

iamchrismiller commented 10 years ago

No Problem.. Let me know if you run into anything.

bboyle commented 10 years ago

I have noticed the same issue on Windows (7 and 8.1). It seems to be an issue with casperjs always having an exit code of 0 on windows. I traced it down to here: https://github.com/bboyle/grunt-casper/blob/master/tasks/lib/casper.js#L30

This post suggests it could be a python behaviour? http://glandium.org/blog/?p=3051 Here's the related line in casperjs: https://github.com/n1k0/casperjs/blob/master/bin/casperjs#L137

I'm not 100% sure though.

ps: I am using windows cmd, not cygwin.

bboyle commented 10 years ago

Good news. Using the latest casperjs from github (master branch) works. Here's the fix: https://github.com/n1k0/casperjs/commit/a67b35d0eb3f96d08476f5bd8b5764ef2246459c

casperjs 1.1 beta 3 (on the casperjs website) was 3 months ago, the above fix was commited 2 months ago.

Thanks for the plugin Chris!