FRSOURCE / cypress-plugin-visual-regression-diff

Perform visual regression test with a nice GUI as help. 💅 Only for Cypress!
MIT License
168 stars 22 forks source link

[Bug]: Long Screenshot (e.g. foto gallery on iphone-8 viewport) is cut/stitched incorrect #270

Closed mduivcap closed 1 year ago

mduivcap commented 1 year ago

Describe the Bug

I'm running tests across multiple browsers and multiple viewports. If the html element you try to match is to big for the screen (needs scrolling I assume), it looks like it's stitched together wrongly. The site itself does not show this issue, when compared Chrome/Edge/Electron with Firefox (in FF it does this correct) image

Steps to reproduce

Have an element (like a photo gallery) that has different layouts on desktop/mobile and that is to long to fit on the screen (scrolling required to view the entire element)

  1. cy.viewport("macbook-13");
  2. cy.get("gallery").matchImage()
  3. cy.viewport("iphone-8");
  4. cy.get("gallery").matchImage()
  5. Open the image of the mobile view and compare that to the real site (just to be sure it's not a real bug)

Expected behavior

The stitch to be correct like in FF image

Additional context

No response

Package version

3.3.10

Device

viewport("iphone-8")

OS and version

Windows 11

Browser and version

chrome (latest), electron (latest), edge(latest) - go wrong ff (latest - does it correctly

Node version

20.4.0

I can work on it by myself

Code of Conduct

mduivcap commented 1 year ago

Closing this issue. Fixed with a cy.wait(). So that means I just need to wait for the right element to be loaded. So adding this before the snapshot did the trick. cy.get(".gallery img").should("be.visible"); But maybe this plugin could wait for everything in the element you're selecting to be loaded first before taking the screenshot

mduivcap commented 1 year ago

Hmm, this didn't resolve my issue completely. the iphone-8 image was still wrong sometimes. However, when I reversed the order (first viewport iphone-8 > match image > viewport macbook-13 > match image), it all went fine. I did try this too, but then the test failed on have.prop naturalWidth https://glebbahmutov.com/cypress-examples/recipes/image-loaded.html So I'm stuck on what else to do to wait properly (since I really don't want a cy.wait(100) in my code).

FRSgit commented 1 year ago

Hm, that seems like a bunch of issues with the cypress itself - in this plugin we're using cypress screenshot api and there is not image stiching happening on our side. It's all cypress - so I'd propose to ask question about that there.

And regarding the checking whether image is loaded or not - I'd expect that the solution using naturalWidth works. And I was just to propose writing a function callback for the should function, so thank you for posting that 😄

Did it resolve all of the issues for you?

mduivcap commented 1 year ago

hmmm, I thought it did, but it doesn't really. I think the issue is that I load a viewport and the gallery is in a normal state (macbook-13), therefore matchImage works fine. Then I change the viewport within the same test (iphone-8) and that means a breakpoint was hit and the gallery should display slightly different and it's still busy with that while matchImage takes the screenshot. Even with this code that does work image has natural width: expected 760 to be above 0, the image is wrong, so I'm not sure how to wait gallery to be done scaling down before taking the screenshot.

cy.get('.gallery img')
  .should('be.visible')
  .and(($img) => {
    // "naturalWidth" and "naturalHeight" are set when the image loads
    expect(
      $img[0].naturalWidth,
      'image has natural width'
    ).to.be.greaterThan(0)
  })

If I create seperate tests for each viewport, all seems to be fine, it's only with the combination within one test. My workaround is to compare the iphone 8 one first and then switch to a bigger viewport. This seems to give stable results so far.

mduivcap commented 1 year ago

Just a quick note. for the viewport I've fixed this with a cy.reload() after changing the viewport. That seemed to do the trick. But in case you need a different solution, you might find it here:: https://github.com/cypress-io/cypress/issues/2681

mduivcap commented 10 months ago

Upon further investigation, I don't think this had anything to do with stitching, but with scrolling. See also: https://github.com/cypress-io/cypress/issues/28674