Nightwatch not recognizizing Chai assertions

[JonDum]

Nightwatch isn't picking up any assertion from Chai (and possibly others).

var expect = require('chai').expect;

module.exports = {
    'Chai expect test':function() {
        expect('dog').to.equal('dog');
    }
}

Results in this output:

[Test] Test Suite
=================

Running:  chai test
 ✔ Passed

----------------------------------------------------
TEST FAILURE: 0 assertions failed, 0 passed (42ms)

If instead I ensure an assertion throws:

Running:  another test
 ✔ Passed
 ✖ Failed
AssertionError: expected 'dog' but got 'cat'
    at Object.module.exports.another test (...)

FAILED:  1 assertions failed (4ms), 0 passed (50ms)

Weirdly it shows ✔ Passed and ✖ Failed at the same time in the reporter haha.

I made a repo to reproduce it and it still happens even on the utmost simple use case. https://github.com/JonDum/nightwatch-chai-error

Tested on latest npm version (0.7.9) via nightwatch -v

[jimmyeisenhauer]

Yeah I had recreated the issue I am seeing in a simple test as well.

https://gist.github.com/jameseisenhauer/c3848a46aed6c913c5cf

Not resulting in a assertion failing being reported. TEST FAILURE: 1 error during execution, 0 assertions failed, 3 passed. (15.684s)

[jimmyeisenhauer]

499 talks about the same thing

[JonDum]

@jameseisenhauer Yea, good catch! I came across those while googling, but the author's description was pretty scatter brained so I wasn't totally sure. I first noticed it while using an execute() call.

'do thing': function(browser) {
   browser
   .execute(function() {
      // do some stuff
      return someThing;
   },
   [],
   function(result) {
       expect(result.value).to.equal(...)
   } 
}

The expect inside the execute callback wasn't working so I thought it was limited to assertions inside nested functions before I realized nightwatch wasn't recognizing them at all.

@beatfactor Was this working at one point or is fixing this going to be new additions? Where would a good place be to start in the code base?

[beatfactor]

The output was working correctly (ie it wasn't showing the Passed when failed). At this point Nightwatch doesnt count the chai assertions so it fails the testcase when something bad happens. That would be a new addition, but I'm not sure that we even can pick up the assertions. The code that handles this is inside lib/runner/testcase.js.

[JonDum]

Ok so I think this might just be confusion with how the reporter... reports.

For example, consider this mocha/chai test:

var expect = require('chai').expect;
describe('Dogs', function() {
    it('should bark', function () {
        // ...
    });
    it('should fetch', function () {
        // ...
    });
});
describe('Cats', function() {
    it('should purr', function () {
        // ...
    });
});

The output of which is a clean and pretty:

  Dogs
    ✓ should bark
    ✓ should fetch

  Cats
    ✓ should purr

  3 passing (8ms)

If I remake the equivalent, entirely passing test suite in nightwatch:

[Cats] Test Suite
=================

Running:  should purr
 ✔ Passed

[Dogs] Test Suite
=================

Running:  should bark
 ✔ Passed

Running:  should fetch
 ✔ Passed

----------------------------------------------------
TEST FAILURE: 0 assertions failed, 0 passed (159ms) 

The bold, bright red TEST FAILURE: 0 assertions failed, 0 passed (159ms) is what I think is getting us confused haha.

Perhaps that should only be shown when there are actual assertions failed, or hidden altogether. I'm comfortable submitting a PR for that.

In contrast to that "passing" test here is a failing test where a simple chai.expect fails:

[Cats] Test Suite
=================

Running:  should purr
 ✔ Passed

[Dogs] Test Suite
=================

Running:  should bark
 ✔ Passed

Running:  should fetch
 ✔ Passed
 ✖ Failed
AssertionError: expected 'dog' to include 'fetch'
    at Object.module.exports.should fetch (/Users/JD/Dropbox/Creative/Projects/nightwatch-chai-error/tests/Dogs.js:11:26)
    at Object.<anonymous> (/Users/JD/Dropbox/Applications/lib/node_modules/nightwatch/lib/util/utils.js:29:8)
    at Module.callAsync (/Users/JD/Dropbox/Applications/lib/node_modules/nightwatch/lib/runner/module.js:65:18)
    at /Users/JD/Dropbox/Applications/lib/node_modules/nightwatch/lib/runner/testcase.js:81:29
    at _fulfilled (/Users/JD/Dropbox/Applications/lib/node_modules/nightwatch/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/Users/JD/Dropbox/Applications/lib/node_modules/nightwatch/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/Users/JD/Dropbox/Applications/lib/node_modules/nightwatch/node_modules/q/q.js:796:13)
    at /Users/JD/Dropbox/Applications/lib/node_modules/nightwatch/node_modules/q/q.js:556:49
    at runSingle (/Users/JD/Dropbox/Applications/lib/node_modules/nightwatch/node_modules/q/q.js:137:13)
    at flush (/Users/JD/Dropbox/Applications/lib/node_modules/nightwatch/node_modules/q/q.js:125:13)

FAILED:  1 assertions failed (2ms)

----------------------------------------------------
TEST FAILURE: 1 assertions failed, 0 passed (161ms)
 ✖ Dogs
   - should fetch

The huge stack trace for an AssertionError is probably unnecessary. I think* all the major assertion libraries specifically use AssertionError so maybe we can check for that with instanceof and not show a stack trace.

[JonDum]

4am and can't sleep.... so naturally I went a bit overboard....

[sharadJay]

any updates on this?

[beatfactor]

Sorry for the lack of updates, I'll look into this soon.

[JonDum]

@beatfactor I rewrote a lot of the logging, but ran into issues because the whole TestRunner expects that if no selenium assertions passed then the test is a failure. It really just needs to assume that if no assertions failed then the test was a success, regardless of how many assertions even ran.

I don't think anyone really cares about how many specific assertions ran in each test, just whether the test passed or failed and why if it failed.

[beatfactor]

Yes, that's a fair point. Will you submit a PR?

[JonDum]

I tried to do it actually, but I think I made a lot of bugs because the concept of a test passing was heavily engrained into the fact that any selenium assertions ran.

If you could fix that I can merge in the pretty logging stuff.

On Thu, Sep 3, 2015 at 10:22 PM, Andrei Rusu notifications@github.com wrote:

Yes, that's a fair point. Will you submit a PR?

— Reply to this email directly or view it on GitHub https://github.com/nightwatchjs/nightwatch/issues/601#issuecomment-137563398 .

[beatfactor]

I can have a look.

[beatfactor]

This has been addressed in 0.8. Feel free to re-open if still unsatisfactory.

[broyeztony]

am running "nightwatch": "^0.8.6" and still have the issue. Chai expectations are passing but the nightwatch reporter does not say it.

[YoungElPaso]

Same. Weird. Might want to update the docs.

[jakejuby]

Same, and would really like this to work.

[vamsys]

running nightwatch 0.9.5, still seing this issue, really be good to make this work, I have a combination of ui and api tests in my framework, and the reporting is a real pain with this

[stanislavgeorgiev]

nightwatch 0.9.12 and still can't make expect to be reported

[jasonkarns]

Same issue that @JonDum recognized 2 years ago(!) is still an issue.

But the handling is slightly unique for execute or executeAsync callbacks. Nightwatch does not catch exceptions thrown by those callbacks so any non-nightwatch assertions (or any other random runtime exception) is completely ignored by nightwatch. Nevermind the question of how nightwatch should report them. At present, it still doesn't even know those exceptions are occurring; let alone finding a helpful way to report on them.

[KSoto]

Still experiencing this problem in v0.9.16. Wrote a StackOverflow question about it: https://stackoverflow.com/questions/47720938/how-do-i-get-nightwatch-to-show-chai-assertions-in-the-reporter