search

notwaldorf / ama

:raising_hand: Ask @notwaldorf anything!
94 stars 13 forks source link

Sparing actual/diff PNGs from failures of Puppeteer screenshot diff tests in CI? #91

Closed anishkny closed 6 years ago

anishkny commented 6 years ago

Hi @notwaldorf !

Loved your post on Automatic visual diffing with Puppeteer. I am trying to setup Puppeteer-screenshot diff tests for my repo vuepress-base with TravisCI.

Any best practices on how to spare actual and diff screenshots in case of failure in TravisCI?

Cheers!

notwaldorf commented 6 years ago

What do you mean by spare? Do you mean see? If so, I haven't found a super good way other than dumping the base64 data as a string, and then displaying it somewhere (with a data:image/png;base64,... tag ). Here is an example of how i dump the actual screenshot, but if you move this to the bottom, you can dump the ${view}-diff.png diff instead).

anishkny commented 6 years ago

Hey thanks for the reply!

Yes, I meant see 😃 . I could figure out how to see the screenshots locally, but if my screenshot diff test fails in CI, how do I see expected/actual/diff PNGs?

The best I could come up with is to upload the PNGs in case of failure to S3/surge.sh or similar. E.g.: http://vuepress-base.surge.sh generated by this code.

Also, I am seeing failures because my golden screenshot is generated on a Mac, but my CI runs it against Linux. And if I adjust the pixelmatch threshold too much, I am afraid the test might silently pass.

Wondering how you handled these situations s'all...