search

Rykus0 / nightwatch-visual-regression

Custom nightwatch assertion and commands for capturing, cropping, and comparing screenshots
MIT License
16 stars 1 forks source link

visualRegression() assertion is ran async(?) instead of sync. #13

Open drptbl opened 8 years ago

drptbl commented 8 years ago

Hey @Rykus0, thanks for your great work last days.

There is one more issues which is the most "disturbing" one - I would say.

It looks like visualRegression() assertion is ran async. It's weird because all nightwatch.js assertions are ran sync. Is there any way to fix this?

For example:

[Home Page / Main Menu Spec] Test Suite
===========================================

Running:  Main menu tests should check that menu exists on home page
ⓘ  Finished setting up everything before testing..
ⓘ  Finished visual regression checks!
No assertions ran.

Running:  another one
 ✖ Visual Regression: Screen differs by 4.19% (see: /diff/Main menu tests should check that menu exists on home page__chrome__v51.0.2704.103__MAC__1680x1050__body.diff.png)  - expected "0.1" but got: "null"
    at Object.module.exports.Main menu tests should check that menu exists on home page (/tests/homePage/mainMenuSpec.js:59:22)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

ⓘ  Finished visual regression checks!

FAILED:  1 assertions failed (2.669s)
 ✖ Visual Regression: Screen differs by 12.55% (see: /diff/another one__chrome__v51.0.2704.103__MAC__1680x1050__body.diff.png)  - expected "0.1" but got: "null"
    at Object.module.exports.another one (/tests/homePage/mainMenuSpec.js:92:24)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

ⓘ  Closing down everything after testing..

 _________________________________________________

 TEST FAILURE:  1 assertions failed, 0 passed. (12.124s)

 ✖ homePage/mainMenuSpec

   - another one (2.669s)
   Visual Regression: Screen differs by 4.19% (see: /diff/Main menu tests should check that menu exists on home page__chrome__v51.0.2704.103__MAC__1680x1050__body.diff.png)  - expected "0.1" but got: "null"
       at Object.module.exports.Main menu tests should check that menu exists on home page (/tests/homePage/mainMenuSpec.js:59:22)
       at _combinedTickCallback (internal/process/next_tick.js:67:7)
       at process._tickCallback (internal/process/next_tick.js:98:9)

In example above: each Finished visual regression checks! log is done after visualRegression(), so it's kind of "weird" that there is no assertion after first log, it's done while running second test case (so it's not sync for sure). Also it says "No assertions ran." after first test case, while visualRegression() was ran there (still issue with async I believe). And summary prints only 1 failure while there was 2 failures actually.

I think that most of this bugs are related to this issues with async issues in visualRegression(); assertion.

Thanks for great work, Cheers!

Rykus0 commented 8 years ago

Yes, this is a good point. When I first wrote it, I followed the examples, which seemed to be all async - which I personally find frustrating more often than helpful. Now that I've worked with it more, I'll see if I can make it synchronous. I'm pretty sure I remember looking into it and finding a way... surprised I hadn't implemented it already.

I'll work on this.

Rykus0 commented 8 years ago

In the meantime, you can use the callback to coordinate, though I know that's annoying.

gregoryduckworth commented 6 years ago

Would love to see this addition!