Pass / Fail status is not shown when tests fail

LesterTheTester commented 10 years ago

Though grunt-nightwatch now sets the exit code correctly, as of 0.1.8, it now does not print the pass/fail status anymore after it runs, like:

✔ Testing if element div.col-sm-4:nth-child(2) contains text: BitTorrent Bundle OK. 1 assertions passed.

Now, when I run the tests, I see no pass/fail status, which is extremely hard to debug in a build, since you'll see something like this, depite tests failing:

Executing "travis" tests GET / 200 6ms GET /vendor/modernizr.js 200 7ms GET /stylesheets/public.b2a037f36765df985f37550d06b4367b.css 200 10ms GET /img/favicon.62a75aa589381f282faead1a788dd423.ico 200 1ms - 1.31kb GET /img/bgd.8c5f105c5c51557897cf85304c2c3833.jpg 200 6ms - 10.95kb GET /javascripts/public/public.f8229475d6e9cd2ae0f2a583b56e7530.js 200 8ms GET /javascripts/base.a107066a56abfed017ec2c8999de2212.js 200 15ms GET /img/logo.d29b8de0953c0e59fbbf1512786a9930.png 200 2ms - 3.47kb POST /api/analytics?action=pageview&label=BundleListPage&category=null&value=0 200 2ms GET /api/bundles?page=%7B%22limit%22%3A1%7D&selectors=%7B%22weight%22%3A100%7D 200 6ms GET /api/bundles?selectors=%7B%22weight%22%3A%7B%22%24ne%22%3A100%7D%7D&page=%7B%22limit%22%3A10%7D&sort=%7B%22acceptedAt%22%3A-1%7D 200 30ms GET /api/bundles?page=%7B%22skip%22%3A0%2C%22limit%22%3A30%7D&selectors=%7B%22weight%22%3A%7B%22%24ne%22%3A100%2C%22%24gte%22%3A20%7D%7D 200 47ms GET /img/ui_icons_sprite.3ffc493f4bcb4dd0a50cad448d3ea494.png 200 2ms - 16.79kb GET /img/bundle-default-background.88d1fc7e8d30d56c5bcc3f7914998025.jpg 200 3ms - 115.35kb

Done, without errors.

Yet, the exit code will be 1: $ echo $? 1

It also seems that tests stop executing once the first test fails. This afternoon someone broke test 2/3 in our tiny grunt-nightwatch suite, and the 3rd test never ran.

Do you have any ideas what's going on here? This behaviour definitely started with 0.1.8.

Thanks,

-Lester

vikki commented 10 years ago

I'm seeing this too - there are actually further problems (at least for me) since runner/run.js has been moved, so I had to edit grunt-nightwatch/tasks/nightwatch.js:171 to point to lib/runner/run.js instead to get anything at all to happen. Once I did that I started seeing the issues you describe, so I think that nightwatch has been refactored a fair bit and no longer passes the error info into the callback from run as it previously did.

I think you could short-term trivially fix it by having the grunt task refer to nightwatch 0.4.7 which seems to work (rather than ~0.4.7 which will get the latest 0.4.* I think, and the more recent 0.4.* build seem to have changed the API so its incompatible), but obviously it'd be nicer to have the task work with the more recent builds.