HuddleEng / PhantomCSS

Visual/CSS regression testing with PhantomJS
MIT License
4.72k stars 259 forks source link

New screenshot generated when expected image already exists #167

Closed fusionfox closed 8 years ago

fusionfox commented 8 years ago

We have a number of screenshot comparison tests using PhantomCSS and they work for us really well, but I've observed that every so often our test run starts generating new expected images with the filename incremented, even though a reference image already exists.

For example, for a long time we had a test that captured the screenshot screenshots/global_nav_0.expected.png, but now every time the test suite is run this is ignored and a new image is created at screenshots/global_nav_2.expected.png even though there are no other images with this filename.

thatblairguy commented 8 years ago

I've encountered this problem myself. It turns out that by default, the snapshot images use the name you provide in the screenshot() call, and then append the index into the overall sequence set of tests. What I suspect is happening to your tests is that initially, "global_nav" was your first test (index 0), and now it's your third (index 2).

Assuming that's your situation, you can change that behavior by setting addIteratorToImage to true when you call phantomcss.init(). The caveats are that you then must supply a filename in the screenshot() call, and the burden is now you to guarantee that fillenames are unique.

fusionfox commented 8 years ago

Seems like you're right; if I run all of my tests it generates a global_nav_3.expected.png but if I run just that test in isolation it generates a global_nav_0.expected.png.

Disappointing but for now I'll take your advice and name the screenshots manually.

thatblairguy commented 8 years ago

FWIW, there's also a fileNameGetter() callback which you can override in the init() call. The example in the README just removes the sequence number, but if need be, you could use that to do something more sophisticated..

jamescryer commented 8 years ago

Yeah, I see this can be a bit annoying - it's there to ensure uniqueness. I would recommend using the fileNameGetter callback for custom naming in non-trivial test suites