search

jaredpalmer / cypress-image-snapshot

Catch visual regressions in Cypress
MIT License
887 stars 160 forks source link

The tests always pass even with differences. #194

Open ricardo-vallejo opened 3 years ago

ricardo-vallejo commented 3 years ago

Actual Behaviour

I have the following configuration:

In the file cypress.json

"env": {
        "failOnSnapshotDiff": true
    }

In the file commands.js

addMatchImageSnapshotCommand({
    capture: 'viewport',
    failureThreshold: 0.05,
    failureThresholdType: 'percent', 
    customDiffConfig: { threshold: 0.1 }
});

When I run the test cases to do a visual regression validation, I get a full success execution even when the images have differences:

(Snapshot Diffs)

┌──────────────────────────────┐ │ Tests: 24
│ Passing: 24
│ Failing: 0
│ Pending: 0
│ Skipped: 0
│ Screenshots: 24
│ Video: false
│ Duration: 25 seconds
│ Spec Ran: palig_sample_spec.js
└──────────────────────────────┘

Expected Behaviour

The test should fail when the image has differences.

Versions Cypress: 7.2 Cypress-image-snapshot: 4.0.1

leschdom commented 3 years ago

I just had the same problem. The implementation here seems unintuitive to me: https://github.com/jaredpalmer/cypress-image-snapshot/blob/v4.0.1/src/command.js#L13

So failOnSnapshotDiff is "truthy" when it is NOT set at all! So doesn't matter if you set it to true, false, 'foobar' - once it has a value the type is no longer undefined.

To get it working, try to remove the 

"env": {
  "failOnSnapshotDiff": true
}

And it should work as expected.

In the case you need to disable this behaviour, do it as described in the README via --env failOnSnapshotDiff=false.

LabSilvana commented 2 years ago

I have this problem and I have not been able to solve it and I do not know why it is happening

dirtydiesel46 commented 1 year ago

Also experiencing this issue

I have removed the --env failOnSnapshotDiff=false from the calling run command and it still passes, I have passed the failOnSnapshotDiff as true, I have set my thresholds to 0.01 percent .

It seems like this package just doesn't fail the CI test at all -

Using cypress 10.9.0 and cypress-image-snapshot 4.0.1

sherinjab commented 1 year ago

I'm facing the same issue and unable to resolve it even after updating en variable as suggested. My understanding was that a new snapshot should be taken only when one does not exist already. However, I notice that if I end up updating the test and run the test again, a new snapshot seems to be created. Also, to see if the test fails I purposely changed the snapshot to have the same name but different to what the test would generate. But in that case, the test once again takes a new snapshot leading the test to pass. Also, noticed another issue, which I assume was created for the same https://github.com/jaredpalmer/cypress-image-snapshot/issues/229

saurabh-canva commented 1 year ago

This seems to be working fine for me @sherinjab. Could you check if you are not passing --env updateSnapshots=true somewhere accidentally ?

Also, what exactly do you mean when you say 'updating the test' ? Can this be any minor update to the test file ?

sedabn commented 1 year ago

Is this solved? Cause I am hitting the same problem. Updating env variable as suggested and setting the threshold to 0.01 didn't help. Did anyone solve this issue?

milu-github commented 7 months ago

I hit the same problem. My tests are always passing even if the images are different. Cypress.Commands.add( "matchImageSnapshotWithRetry", (snapshotname, options = {}) => { options.timeout = 0; options.delayBetweenTries = 0; options.failureThreshold = 0.01; options.failureThresholdType= 'percent', cy.matchImageSnapshot(snapshotname, options); } );

These are the options iam passing to the matchImageSnapshot command.

I can see logs in runner that cypress-image-snapshotImage was 2.3626666666666667% different from saved snapshot with 24365 different pixels. But still my tests are passed. I am using "@emerson-eps/cypress-image-snapshot": "^1.5.2". Can someone tell me the solution for this?

simonsmith commented 7 months ago

@milu-github This repository is no longer maintained. You're using a fork that was taken from https://github.com/simonsmith/cypress-image-snapshot but is quite different judging by the commit differences.

I'd recommend opening an issue against @emerson-eps/cypress-image-snapshot