search

jaredpalmer / cypress-image-snapshot

Catch visual regressions in Cypress
MIT License
888 stars 160 forks source link

Failing tests don't fail if you retry them. #156

Open VicJer opened 3 years ago

VicJer commented 3 years ago

I have came across an issue hopefully someone can enlighten me. My test is pretty much an one liner and uses the settings from the docs and it works great but when combined with the cypress setting for automatic retries eg retries:{runMode:2} it fails on the first try then any consecutive retry passes. I narrowed it down to the fact that snapshot name changes and the test generates a new one on each retry with (attempt X) in the name. I am assuming the comparison method picks up that new screenshot instead of the one it already has. My workaround was to create multiple copies of the screenshot with appropriate attempt suffix but it feels not right.

as regards the test it literally is cy.get('#viewer').matchImageSnapshot( 'testSnapshot', options) and options are 

{
    comparisonMethod: 'ssim',
    failureThreshold: 0.1,
    failureThresholdType: 'percent',
};
lmeikle commented 3 years ago

Hey Wojciech, I had the same issue and raised a PR for it last week https://github.com/jaredpalmer/cypress-image-snapshot/pull/155

VicJer commented 3 years ago

Hey Wojciech, I had the same issue and raised a PR for it last week #155

Oh awesome that is EXACTLY what I need 👍

isaac-jordan commented 3 years ago

This can be resolved now that 4.0.1 has been released with the fix.