Comparison issues in 4.5.0, probably arising from the upgrade to puppeteer v2

[deleemillie]

We have some 450 backstop tests that run against a library, and recently 444 of those started failing with no changes. The changes being shown were all similar to the below.

Either the fonts were being rendered differently (the screenshot taken on the right came from the most recent 4.5.0 upgrade, the left was 4.4.2): https://ibb.co/16vt4dH

Or snapshots were being randomly cut off (sometimes what we wanted snapshot-ed did not even end up in the image; again, the image that came from 4.5.0 is to the right): https://ibb.co/DQcCm0s

Here is an example of the configuration we are using:

{
   "label":"Generic Components - Buttons - Button",
   "url":"http://localhost:6060/#/Generic%20Components/Buttons/Button",
   "selectorExpansion":true,
   "selectors":[
      ".styleguideist-component-wrapper"
   ],
   "misMatchThreshold":0.05,
   "scrollToSelector":".styleguide-main-wrapper",
   "readySelector":".styleguideist-component-wrapper",
   "postInteractionWait":300,
}

Here are the backstop settings:

id: 'backstop_default',
  viewports: [
    {
      label: 'desktop',
      width: 1920,
      height: 1080,
    },
  ],
  onReadyScript: 'onReady.js',
  scenarios,
  paths: {
    bitmaps_reference: 'testing/ui/bitmaps-reference',
    bitmaps_test: 'testing/ui/bitmaps-test',
    engine_scripts: 'testing/ui/engine-scripts',
    html_report: 'testing/ui/html-report',
    ci_report: 'testing/ui/ci-report',
  },
  report: ['browser'],
  engine: 'puppeteer',
  engineOptions: {
    args: ['--no-sandbox'],
  },
  // Helps resolve issue with misaligned screenshots
  puppeteerOffscreenCaptureFix: true,
  asyncCaptureLimit: 5,
  asyncCompareLimit: 50,
  debug: false,
  debugWindow: false,

Each url has several blocks we want snapshot-ed separately, hence why the selector expansion. Variations to delay etc have no effect in resolving the problem.

I figured maybe the font differences are as a result of the rendering engine changing, which is fine and we could update our snapshots to match (though they no longer really appear the same as what Chrome renders). But the trimmed snaphots are a greater problem.

Previously when I had that problem, the puppeteerOffscreenCaptureFix: true, setting fixed it. Now, regardless of settings, the issue persists. We're not doing anything custom in our ready scripts.

At this stage I'm unsure what to change in order to make this work. Thank you!

[garris]

Are you using the docker option?

[deleemillie]

@garris no, we are not using the docker option.

[verpixelt]

Having the same issue over here with a non-dockerized version. Sometimes ending up with cut off images, reference and test.