Backstop on GH action – when I am running backstop on the gh action, it wont detect the references

[paschalidi]

Heya,

thank you for this great project.

I am having my storybook, which looks like this https://ever-storybook-refs-pull-190-merge.surge.sh/ And also I have this gh action

name: tests - visual regression

on:
  pull_request:
    branches:
      - '**'

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 21
          cache: 'npm'

      - name: Install dependencies
        run: |
          npm i

      - name: Build storybook
        run: npm run build-storybook

      - name: Deploy storybook to Surge
        run: |
          BRANCH_NAME=$(echo ${{ github.ref }} | sed 's/refs\/heads\///g' | sed 's/\//-/g')
          DOMAIN="ever-storybook-${BRANCH_NAME}.surge.sh"
          npx surge --project storybook-static --domain $DOMAIN --token ${{ secrets.SURGE_TOKEN }}
          echo "STORYBOOK_URL=https://$DOMAIN" >> $GITHUB_ENV

      - name: Run Visual Regression Tests
        run: |
          node ./backstop_data/backstop-init.mjs ${{ env.STORYBOOK_URL }}
          cat ./backstop_data/backstop.json
          ls -la backstop_data
          ls -la backstop_data/bitmaps_reference
          npx playwright install 
          npm run backstop:test

      - name: Upload backstop_data artifact
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: backstop_data
          path: backstop_data/

      - name: Check bitmaps_test directory
        if: failure()
        run: ls -la backstop_data/bitmaps_test

      - name: Deploy Backstop.js report to Surge
        if: failure()
        run: |
          BRANCH_NAME=$(echo ${{ github.ref }} | sed 's/refs\/heads\///g' | sed 's/\//-/g')
          DOMAIN="ever-visual-regression-${BRANCH_NAME}.surge.sh"
          npx surge backstop_data/html_report $DOMAIN --token ${{ secrets.SURGE_TOKEN }}
          echo "Backstop.js report deployed to: https://$DOMAIN"

The problem is that when I run the tests locally the report I am getting is correct and the test are passing correctly.

When though I run the same tests on my gh action it seems that it wont pick up the references. The html report from the tests looks like this https://ever-visual-regression-refs-pull-190-merge.surge.sh/

I am uploading also the backstop_data folder. backstop_data (1).zip

It seems to me that all the data is there, the reference images and the test images are there but for some reason it wont pick it up in the gh action env. Could u please help me understand what am I missing here?

[dgrebb]

I know this one! The report expects to be served from a non-root path. Something like /backstop_data/html_report/index.html. The tests (and reference images) are actually working, but the images are missing from the UI.

The regression site is throwing 404s for the images.

Double check the paths... where are those images actually located in surge.sh's filesystem?

It may be easiest to change the action to upload the full path, or use something like patch-package to change what the reporter generates for the images.

I made a similar hack for backstop remote, via the express server.

Hope this helps. Cheers!

[paschalidi]

Thank you @dgrebb I will be trying that one today 🙇

[paschalidi]

Reporting that this worked, as you very well said, I wasnt including the backstop_data files. Now for reference I am doing the following

name: tests - visual regression

on:
  pull_request:
    branches:
      - '**'

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 21
          cache: 'npm'

      - name: Install dependencies
        run: |
          npm i

      - name: Build storybook
        run: npm run build-storybook

      - name: Deploy storybook to Surge
        run: |
          BRANCH_NAME=$(echo ${{ github.ref }} | sed 's/refs\/heads\///g' | sed 's/\//-/g')
          DOMAIN="ever-storybook-${BRANCH_NAME}.surge.sh"
          npx surge --project storybook-static --domain $DOMAIN --token ${{ secrets.SURGE_TOKEN }}
          echo "STORYBOOK_URL=https://$DOMAIN" >> $GITHUB_ENV

      - name: Run Visual Regression Tests
        run: |
          node ./backstop_data/backstop-init.mjs ${{ env.STORYBOOK_URL }}
          cat ./backstop_data/backstop.json
          ls -la backstop_data
          ls -la backstop_data/bitmaps_reference
          npm run backstop:test

      - name: Upload backstop_data artifact
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: backstop_data
          path: backstop_data/

      - name: Check bitmaps_test directory
        if: failure()
        run: ls -la backstop_data/bitmaps_test

      - name: Deploy Backstop.js report to Surge
        if: failure()
        run: |
          BRANCH_NAME=$(echo ${{ github.ref }} | sed 's/refs\/heads\///g' | sed 's/\//-/g')
          DOMAIN="ever-visual-regression-${BRANCH_NAME}.surge.sh"
          cp -r backstop_data/html_report/* backstop_data/ // <- this is what I added 
          npx surge backstop_data $DOMAIN --token ${{ secrets.SURGE_TOKEN }}
          echo "Backstop.js report deployed to: https://$DOMAIN"