simonsmith / cypress-image-snapshot

Catch visual regressions in Cypress with jest-image-snapshot
MIT License
65 stars 15 forks source link

snapshot comparison is weird #49

Closed maximegheraille closed 7 months ago

maximegheraille commented 8 months ago

Hi,

I am wondering if I am missing something or why this is happening.

I created my snapshots like I always do cy.get(element).scrollIntoView(); cy.get(element).matchImageSnapshot(options)

they weird thing is that I added a component before the one for which I take snapshots and suddenly some snapshots are failing, with a diff between 1% and 11%. The diff snapshots do show diffs but honesty none of them have diffs that I even can see with my eyes.

Here is an example. diff

I don't see why this one and few others are randomly now failing (the component itself did not change), if I remove the component added before this one it never fails again.

Not sure why adding that component makes the snapshots fail.

simonsmith commented 8 months ago

What settings did you use to achieve that snapshot result?

At first glance it looks like something has shifted the layout, perhaps inserted elsewhere

maximegheraille commented 8 months ago

hey,

these are my setting, I also initially thought that there were layout shifts but there is nothing in that component that moves. Did you ever try something like this (moving a component other than the one for the snapshot) ?

const options: CypressImageSnapshotOptions = {
            failureThreshold: 0.01,
            allowSizeMismatch: true,
            failureThresholdType: 'percent',
            capture: 'viewport',
            comparisonMethod: 'ssim',
            customSnapshotsDir: `./cypress/reports/mochareports/snapshots/something`,
            e2eSpecDir: Cypress.spec.relative,
            snapFilenameExtension: '.snap',
        };
simonsmith commented 8 months ago

It seems like your comparison image is lacking detail. For example the defaults look like this:

image

That would help you identifiy the cause I imagine

maximegheraille commented 8 months ago

hi, that would indeed already help, but I don't really see how I can improve them, everything is run in a docker via Github actions. do you have any ideas ?

simonsmith commented 8 months ago

Is it possible to reproduce in some way? Not much I can do from here otherwise