Image size mismatch on all output

SlexAxton commented 10 years ago

I create screenshots when developing on OS X, but my CI server is Ubuntu. Things seem to be going alright, except all the screenshots that are being taken on Ubuntu are 1 pixel less tall than they should be.

Error: Images not the same dimension. First: 1200x479. Second: 1200x480.

No matter which dimensions I choose, this seems to be the case.

Using selenium 2.42.2 and firefox

SlexAxton commented 10 years ago

I can also verify this happening with chrome as well, but seemingly way different sizes (likely due to the 'flags' warning up at the top of chrome):

Error: Images not the same dimension. First: 1200x455. Second: 1200x444

chenglou commented 10 years ago

What Instagram does is that they configure Huxley as part of their remote CI (for node-huxley, here's an example API usage: https://github.com/chenglou/grunt-huxley/blob/master/tasks/huxley.js#L16-L22). So there's no local screenshot capturing here. I think that's what lots of people using this do. Because really, switching OS and display and hoping for screenshots to be consistent is akin to switching browser and hope for the same thing: it's not always perfect. I've considered setting an error threshold for image comparison, but that's risky since there'd be false positives.

But I also don't think sending everything to the server is a nice UX. The whole point is to be able to see your replay. I've been thinking about this recently and I think I came up with a balanced solution: #65. Discuss the idea there if you want.

SlexAxton commented 10 years ago

The more I worked at it today, the more I agree that the multiplatform thing just can't work right now. I think something like saucelabs or browserstack make this possible, though. I can configure my local box to take screenshots and then when i send the code to CI, it runs against the saucelabs webdriver endpoint, against a computer with the same OS and browser that I use. You could also run against additional browsers in CI only (mobile devices, etc).

I think #65 works for a lot of scenarios, but I can think of a handful of times in the last week that I'd need to do much crazier things than just a stash to get old code in a working state. I think I'd vote for more support of multi-device/multi-OS screenshots via sauce/browserstack before I voted for the stash tricks. But by all means, you have much more context than me, and I trust your decisions!

chenglou commented 10 years ago

Well, stash was just an example of nice default when the process is automated. In fact, you can manually do that process almost today:

git stash # or whatever that reverts to the correct state
# imaginary huxley option
hux -u --dest-folder=huxleyScreenshots_local
git unstash
hux --dest-folder=huxleyScreenshots_local
# ...when everything matches
rm -r huxleyScreenshots_local
git commit

So the screenshots you want to run on the server are still in a legit "/huxleyScreenshots" folder against which you can compare in CI. For teams who don't use saucelab/remote CI/wanna see the result pass before pushing, the local option would be suitable.

Does that make sense? If it does I'll think of a way to make this pain-free. I'm also sick of pixel-rounding errors.

chenglou commented 10 years ago

https://github.com/chenglou/node-huxley/blob/master/HISTORY.md#080 =)

chenglou / node-huxley

Image size mismatch on all output #64