Diff tolerance

[jhnns]

What do you think about tolerances when a diff is considered to be a failure?

In my situation I'm working with css transforms and it seems like there are "ghost diffs" maybe caused by sub-pixel rendering issues.

[chenglou]

I am thinking of that! I didn't have time to look into this too much but I took the precaution to isolate this into a library: https://github.com/chenglou/png-diff/issues/2 If you feel like tackling, go ahead =). If not, I'll see how this is done.

[jhnns]

Why did you decide to use a js-based solution? There are great image-diff libraries out there.

I wonder which approach would be the best for our use-case. I think a small bias per pixel value is ok, but a great bias should be accumulated until a given threshold.

[chenglou]

Huxley used to use GraphicsMagick, but I had some ease-to-use constraints that made me drop it and opt for a native-less solution instead. At one point, I even thought of bundling selenium (#13) to avoid questions like this.

In retrospect, it's still been a helpful decision for designers and alike, but not for me. I reinvented the wheel and did worse =).

If the RMSD solution isn't too complicated then that might be the way to go; Or else I'm fine with reincluding GraphicsMagick as a dep.

[jhnns]

I like the approach of the phantomjs-package: It just downloads the appropriate binary release and provides a path so it can easily be spawned inside node via a sub_process. It's a good compromise between easy-to-use and native libs. It yielded to great libs which rely on phantomjs.

I don't think that the RMSD solution is too complicated but I'm not sure whether the algorithm will produce satisfactory results.

I'll probably tackle this when I still get these ghost diffs.

[chenglou]

I guess I could close this for now, as 0.8.0 completely side-steps this problem by comparing screenshots locally.

[jhnns]

Interesting. How does that work?

(I'm glad that you could make some progress)

[chenglou]

See the first paragraph here: https://github.com/chenglou/node-huxley/wiki/Advanced-usage

Simple really =)

[jhnns]

Using git? ^^

[chenglou]

Yep. I'll implement hg if there's demand. If you're not using version control at all (why, lol), the previous writeScreenshots and compareScreenshots are still available to you.

[jhnns]

That's clever (yes, I'm using git) :+1:

However, it's basically the same as before because git also just compares on byte level. This means that comparing screenshots can't be automated (which is fine for the moment, because it's not a trivial task).

[chenglou]

because git also just compares on byte level

Maybe you misunderstood it, but git's only there for me to stash your changes and take previous screenshots. If there are any inconsistencies in the screenshots taking, they're be the same ones before and after stashing, so the diff errors cancel each other out.

As for automating screenshots, what do you mean by that?

[jhnns]

Ahhh now I get it.

But what happens if I didn't change a task, but just code which affects the screenshots?

[chenglou]

The screenshots before your code change will look different than the screenshots after the change. Here's the example repo: https://github.com/chenglou/huxley-example. git clone that and and change some stuff in test_page.html. Should only take 30 secs =).

[jhnns]

That is clever :grin:... So if huxley says that there is a diff, everything might be ok when the change is intended.

However, it's good that there is still the write mode because it's also good to have a static representation of how the website should look in different situations.

[chenglou]

Yeah. Huxley just stores the folders whose screenshots have a diff (cleans the others. It's a good default I think). What you want to do with those folders is up to you now; most probably check whether the difference is intended (then manually delete the folder yourself).

And yes, one interesting benefit of writing "unit tests" in Huxley is that, assuming you have a good coverage, you can revert your repo to any point in time, hux --write, and see what all your components looked like =).

So really, the "big change" is really the fact that I prescribed a new, optional, local workflow. You can still go back to the old paradigm, check those screenshots in version control, and run everything remotely and whatnot. It's just not recommended anymore because of those screenshot diffs problems.

[chenglou]

One more thing: the default workflow is approximately: git stash && hux --write && git unstash && hux --compare && rm -rf ./*.hux

Except it does the repo stashing and reverting correctly (pretty important lol) and gets all the unchanged (https://github.com/chenglou/node-huxley/blob/60e7218604af946ae222415b59783fa3d8c1b1dd/source/defaultWorkflow/getUnchanged.js#L16-L17) Huxleyfiles and record files to run (so that it doesn't try to run an inexistent Huxleyfile)

Actual code: https://github.com/chenglou/node-huxley/blob/60e7218604af946ae222415b59783fa3d8c1b1dd/source/defaultWorkflow/defaultWorkflow.js

So in the future, I could easily expose an option to compare two arbitrary commits. To visualize what changes you've made to your components across time. Food for thought =).

[jhnns]

Pretty neat! :grinning: