garris / BackstopJS

Catch CSS curve balls.
http://backstopjs.org
MIT License
6.66k stars 602 forks source link

Issues running in GitHub actions #1522

Closed daryl-sf closed 7 months ago

daryl-sf commented 7 months ago

Hi,

I'm trying to run backstop test as part of CI using GitHub actions. When I do there are errors generating the test images. It looks like this is related to a dependency of backstop. Has anyone come across this issue or has had success running backstop as GitHub action?

Stack trace

Disposing Browser
      COMMAND | Executing core for "report"
node:events:495
      throw er; // Unhandled 'error' event
      ^

Error: Invalid file signature
    at Parser._parseSignature (/home/runner/work/component-library/component-library/node_modules/@mirzazeyrek/node-resemble-js/node_modules/pngjs/lib/parser.js:48:18)
    at ChunkStream._processRead (/home/runner/work/component-library/component-library/node_modules/@mirzazeyrek/node-resemble-js/node_modules/pngjs/lib/chunkstream.js:160:13)
    at ChunkStream._process (/home/runner/work/component-library/component-library/node_modules/@mirzazeyrek/node-resemble-js/node_modules/pngjs/lib/chunkstream.js:175:14)
    at ChunkStream.write (/home/runner/work/component-library/component-library/node_modules/@mirzazeyrek/node-resemble-js/node_modules/pngjs/lib/chunkstream.js:57:8)
    at PNG.write (/home/runner/work/component-library/component-library/node_modules/@mirzazeyrek/node-resemble-js/node_modules/pngjs/lib/png.js:95:16)
    at ReadStream.ondata (node:internal/streams/readable:809:22)
    at ReadStream.emit (node:events:517:28)
    at addChunk (node:internal/streams/readable:368:12)
    at readableAddChunk (node:internal/streams/readable:341:9)
    at Readable.push (node:internal/streams/readable:278:10)
Emitted 'error' event on  instance at:
    at module.exports.emit (node:events:517:28)
    at ChunkStream._process (/home/runner/work/component-library/component-library/node_modules/@mirzazeyrek/node-resemble-js/node_modules/pngjs/lib/chunkstream.js:187:10)
    at ChunkStream.write (/home/runner/work/component-library/component-library/node_modules/@mirzazeyrek/node-resemble-js/node_modules/pngjs/lib/chunkstream.js:57:8)
    [... lines matching original stack trace ...]
    at Readable.push (node:internal/streams/readable:278:10)
    at node:internal/fs/streams:277:14

config

const baseUrl = process.env.STORYBOOK_URL || "http://localhost:6006";
const projectId = "Visual testing";

var storiesContainer = require("./stories.json");
var stories = Object.values(storiesContainer.stories);

const lightmode = stories
  .filter((story) => story.name !== "Docs")
  .map((story) => ({
    path: `/iframe.html?viewMode=story&id=${story.id}`,
    label: `${story.title} - ${story.name}`,
  }));

const darkmode = stories
  .filter((story) => story.name !== "Docs")
  .map((story) => ({
    path: `/iframe.html?viewMode=story&id=${story.id}&globals=theme:dark`,
    label: `${story.title} - ${story.name} - Dark`,
  }));

const relativeUrls = [...lightmode, ...darkmode];

const viewports = [
  {
    label: "desktop",
    width: 1280,
    height: 1024,
  },
];

/**
 * Override the delay for specific components
 * This is useful for components that have animations
 * that need to be completed before taking a screenshot.
 * The delay is in milliseconds.
 * Labels can be partial matches so groups of components can be targeted.
 */
const delayOverrides = [
  {
    label: "Components/Chart",
    delay: 2000,
  },
];

const scenarios = relativeUrls.map((relativeUrl) => {
  const DELAY =
    delayOverrides.find((delayOverride) =>
      relativeUrl.label.includes(delayOverride.label),
    )?.delay || 500;
  return {
    label: relativeUrl.label,
    url: `${baseUrl}${relativeUrl.path}`,
    delay: DELAY,
    readyEvent: "ready-for-screenshot",
  };
});

module.exports = {
  id: projectId,
  viewports,
  scenarios,
  onReadyScript: "playwright/onReady.cjs",
  onBeforeScript: "playwright/onBefore.cjs",
  paths: {
    bitmaps_reference: "backstop_data/bitmaps_reference",
    bitmaps_test: "backstop_data/bitmaps_test",
    html_report: "backstop_data/html_report",
    ci_report: "backstop_data/ci_report",
    engine_scripts: "backstop_data/engine_scripts",
  },
  report: ["json"],
  engine: "playwright",
  engineOptions: {
    args: ["--no-sandbox"],
  },
  asyncCaptureLimit: 5,
  asyncCompareLimit: 50,
  debug: false,
  debugWindow: false,
};
daryl-sf commented 7 months ago

After further investigation I found this issue was caused with using git-lfs to store reference files. Storing files directly works as expected.

edit: It also works if you call git lfs pull before running backstop. This will fetch and checkout the actual images to the filesystem