search

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
47.05k stars 3.19k forks source link

blackout for screenshots does not accurately calculate blackout area #5842

Closed jennifer-shehane closed 1 year ago

jennifer-shehane commented 4 years ago

Current behavior:

This issue is 2 fold, but overall demonstrates that the algorithm for calculating the blackout area of a screenshot is not thorough enough to cover all situations.

  1. When specifying a DOM element to blackout of screenshots, the blackout area is not accurately calculated when run on Chrome versions below 78.0.3904.108.
  2. The blackout area is not accurately calcuated at different viewport sizes (Chrome 78).

1. Chrome versions < 78

Below, the screenshot should black out the red area, but as you can see in Chrome 77 it is not.

I also tested this back to version Cypress 3.2.0 and the issues exhibits on all versions of Cypress.

Chrome 77.0.3865.90

blackout-cy-3 7 0-chrome-77 0 3865 90

Chrome 78.0.3904.108

blackout-cy-3 7 0-chrome-78 0 3904 108

2. Viewport size calculations

Below, the screenshot should black out the red area, but as you can see in smaller viewport sizes it is not.

1400x1080

blackout-1400x1080

1000x660

blackout-1000x660

850x500

blackout-850x500

450x200

blackout-450x200

Desired behavior:

The DOM element specified should be blacked out on all Chrome versions at all viewport sizes

Steps to reproduce: (app code and test code)

I tried several variations of content, removing the lorem ipsum (so that scroll happens) will make the blackout not work or work under certain circumstances, but the below example will fail in all examples specified above.

index.html

<html>
<body>
  <p>Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec ullamcorper nulla non metus
  auctor fringilla. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Lorem ipsum dolor sit
  amet, consectetur adipiscing elit. Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Aenean eu leo
  quam. Pellentesque ornare sem lacinia quam venenatis vestibulum.

  Nullam id dolor id nibh ultricies vehicula ut id elit. Donec ullamcorper nulla non metus auctor fringilla. Cras justo
  odio, dapibus ac facilisis in, egestas eget quam. Donec id elit non mi porta gravida at eget metus. Nulla vitae elit
  libero, a pharetra augue. Maecenas sed diam eget risus varius blandit sit amet non magna. Maecenas faucibus mollis
  interdum.

  Nulla vitae elit libero, a pharetra augue. Praesent commodo cursus magna, vel scelerisque nisl consectetur et. Donec sed
  odio dui. Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor.

  Cras mattis consectetur purus sit amet fermentum. Donec ullamcorper nulla non metus auctor fringilla. Cras justo odio,
  dapibus ac facilisis in, egestas eget quam. Etiam porta sem malesuada magna mollis euismod. Sed posuere consectetur est
  at lobortis.Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec ullamcorper nulla non metus
  auctor fringilla. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Lorem ipsum dolor sit
  amet, consectetur adipiscing elit. Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Aenean eu leo
  quam. Pellentesque ornare sem lacinia quam venenatis vestibulum.

  Nullam id dolor id nibh ultricies vehicula ut id elit. Donec ullamcorper nulla non metus auctor fringilla. Cras justo
  odio, dapibus ac facilisis in, egestas eget quam. Donec id elit non mi porta gravida at eget metus. Nulla vitae elit
  libero, a pharetra augue. Maecenas sed diam eget risus varius blandit sit amet non magna. Maecenas faucibus mollis
  interdum.

  Nulla vitae elit libero, a pharetra augue. Praesent commodo cursus magna, vel scelerisque nisl consectetur et. Donec sed
  odio dui. Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor.

  Cras mattis consectetur purus sit amet fermentum. Donec ullamcorper nulla non metus auctor fringilla. Cras justo odio,
  dapibus ac facilisis in, egestas eget quam. Etiam porta sem malesuada magna mollis euismod. Sed posuere consectetur est
  at lobortis.Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec ullamcorper nulla non metus
  auctor fringilla. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Lorem ipsum dolor sit
  amet, consectetur adipiscing elit. Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Aenean eu leo
  quam. Pellentesque ornare sem lacinia quam venenatis vestibulum.

  Nullam id dolor id nibh ultricies vehicula ut id elit. Donec ullamcorper nulla non metus auctor fringilla. Cras justo
  odio, dapibus ac facilisis in, egestas eget quam. Donec id elit non mi porta gravida at eget metus. Nulla vitae elit
  libero, a pharetra augue. Maecenas sed diam eget risus varius blandit sit amet non magna. Maecenas faucibus mollis
  interdum.

  Nulla vitae elit libero, a pharetra augue. Praesent commodo cursus magna, vel scelerisque nisl consectetur et. Donec sed
  odio dui. Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor.
  </p>
  <div width="560" height="315" style="background-color: red;">Nullam id dolor id nibh ultricies vehicula ut id elit. Donec ullamcorper nulla non metus auctor fringilla. Cras justo
  odio, dapibus ac facilisis in, egestas eget quam. Donec id elit non mi porta gravida at eget metus. Nulla vitae elit
  libero, a pharetra augue. Maecenas sed diam eget risus varius blandit sit amet non magna. Maecenas faucibus mollis
  interdum.

  Nulla vitae elit libero, a pharetra augue. Praesent commodo cursus magna, vel scelerisque nisl consectetur et. Donec sed
  odio dui. Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor.

  Cras mattis consectetur purus sit amet fermentum. Donec ullamcorper nulla non metus auctor fringilla. Cras justo odio,
  dapibus ac facilisis in, egestas eget quam. Etiam porta sem malesuada magna mollis euismod. Sed posuere consectetur est
  at lobortis.Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec ullamcorper
  nulla non metus
  auctor fringilla. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Lorem ipsum dolor sit
  amet, consectetur adipiscing elit. Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor. Aenean eu leo
  quam. Pellentesque ornare sem lacinia quam venenatis vestibulum.

  Nullam id dolor id nibh ultricies vehicula ut id elit. Donec ullamcorper nulla non metus auctor fringilla. Cras justo
  odio, dapibus ac facilisis in, egestas eget quam. Donec id elit non mi porta gravida at eget metus. Nulla vitae elit
  libero, a pharetra augue. Maecenas sed diam eget risus varius blandit sit amet non magna. Maecenas faucibus mollis
  interdum.

  Nulla vitae elit libero, a pharetra augue. Praesent commodo cursus magna, vel scelerisque nisl consectetur et. Donec sed
  odio dui. Vivamus sagittis lacus vel augue laoreet rutrum faucibus dolor auctor.</div>
</body>
<script>
</script>
</html>

spec.js

it('blackout', () => {
  cy.viewport(1400, 1080) // play with viewport sizes here
  cy.visit('index.html')
  cy.screenshot('blackout', { blackout: ['div'] })
})

Versions

Cypress 3.2.0 - 3.7.0 (all tested versions) Chrome 77.0.3865.90

dploeger commented 4 years ago

I can confirm, that this also happens on Electron (found it on Electron 80 and Cypress 4.3.0)

bkfarnsworth commented 4 years ago

@jennifer-shehane should we remove this functionality from the docs until it is fixed? I think its causing a lot of people to spend time debugging when it is a known issue.

Or at least add a note to the docs about this bug. And I'm happy to submit a PR with a doc change if you think that is a good idea.

j1000 commented 4 years ago

That's some impressive issue documentation. I'll add that the misalignment is exacerbated when the test is not run headless. It also happens in Chrome 85, Cypress 5.2.0.

During non-headless playback, the degree of misalignment is influenced by:

I'm especially interested in this fix because I'm using blackout as a shortcut for visual testing. Black out the non-deterministic bits of the screen, take the shot, and move on. Consistent function of the blackout option is crucial for this.

ghost commented 3 years ago

I am seeing this issue with Cypress 6.2.0 using Electron 87 and Chrome 89. I am trying to use the functionality for visual testing.

wplucinsky commented 3 years ago

This issue is also present in Firefox 78 with Cypress 6.0.1

archfz commented 2 years ago

I can confirm this issue. Most of the elements I try it with the black box is not properly covering them. With some elements it's not even working, for example my position fixed header is not blacking out at all, which is for which I would use this feature most.

Now I tried hiding the elements manually before taking the screenshot. But I realized cypress opens a new window or something like that to take the screenshot, thus this hack is not working.

My questions is why is there an algorithm to calculate such a black box positions? Why not just take the DOM snapshot, add a new div inside the target elements with position absolute, width height 100%, and background black; and after that take the screenshot? @jennifer-shehane

archfz commented 2 years ago

Actually never mind. I have found the following to work pretty well as an alternative:

Cypress.Commands.add('withHidden', (selector, callback) => {
  cy.get(selector).invoke('attr', 'style', 'display: none');
  callback();
  cy.get(selector).invoke('attr', 'style', '');
});
cy.withHidden('header', () => cy.get('.element-to-captur').screenshot());
viktorgogulenko commented 2 years ago

Any updates here? Seems making screenshots with Cypress is a big mess - we are trying to use it for our snapshot testing and with each build we are receiving different image sizes even if nothing was changed in code and the same Docker container was used.

cypress-app-bot commented 1 year ago

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

cypress-app-bot commented 1 year ago

This issue has been closed due to inactivity.