Getting websocket error with @web/test-runner-visual-regression

slalomzacharyd commented 4 months ago

I'm trying to sue the visual regression plugin, but when I add it to my code I get websocket errors in my tests. Thank you.

Console output

~/Development/project% ./node_modules/.bin/web-test-runner test/components/test.js

test/components/test.js:

 test [Chrome]
   𐄂 works

 ❌ test > works
      Error: Could not setup web socket connection. Are you executing this test through Web Test Runner?
        at executeServerCommand (node_modules/@web/test-runner-commands/browser/commands.mjs:23:10)
        at async visualDiff (node_modules/@web/test-runner-visual-regression/browser/commands.mjs:28:19)
        at async n.<anonymous> (src/components/test.tsx:9:8)

Test

import Component from '.';
import { visualDiff } from '@web/test-runner-visual-regression';
import { render } from 'testUtils';

describe('Component', () => {
    it('works', async () => {
        const element = await render(<Component />);

        await visualDiff(element.container, 'div');
    });
});

command

./node_modules/.bin/web-test-runner

Packages

    "@web/dev-server-esbuild": "1.0.2",
    "@web/dev-server-import-maps": "0.2.0",
    "@web/dev-server-rollup": "0.6.1",
    "@web/test-runner": "0.18.0",
    "@web/test-runner-commands": "0.9.0",
    "@web/test-runner-junit-reporter": "0.7.0",
    "@web/test-runner-puppeteer": "0.16.0",
    "@web/test-runner-visual-regression": "0.9.0",

configuration

import { defaultReporter, summaryReporter } from '@web/test-runner';
import { visualRegressionPlugin } from '@web/test-runner-visual-regression/plugin';
import { config } from 'dotenv';
import { globSync } from 'glob';

// const commonjs = fromRollup(rollupCommonjs);

config();

const files = globSync('test/**/*.test.js');

export default {
  coverageConfig: {
    reporters: ['cobertura', 'lcov'],
    threshold: {
      branches: 70,
      functions: 70,
      lines: 70,
      statements: 70,
    },
  },
  files,
  plugins: [
    visualRegressionPlugin({
      // update: process.argv.includes('--update-visual-baseline'),
      update: true,
    }),
  ],
  reporters: [summaryReporter(), defaultReporter()],
  testRunnerHtml: testFramework =>
    `<html>
    <head>
    <style>
    .mapboxgl-canary {
      background-color: salmon;
    }
    </style>
    </head>
      <body>
        <script>window.process = { env: { NODE_ENV: "development" } }</script>
        <script type="module" src="${testFramework}"></script>
      </body>
    </html>`,
};

slalomzacharyd commented 4 months ago

I've investigated the defaults a bit and it looks like the default html loads a websocket module which isn't documented. I'm going to try injecting it to see if it fixes the problem. (Though there seems to be some other custom scripting as well.)

![Uploading Screenshot 2024-03-08 at 17.58.00.png…]()

slalomzacharyd commented 4 months ago

Hrm... seems like it is being injected with the custom testFramework. No idea why websockets woudn't work yet.

slalomzacharyd commented 4 months ago

I finally figured out the reason it was failing was because I was bundling it using esbuild (external to web test runner.) My guess is that this library might be relying on globals. Is bundling worth supporting with this plugin?

modernweb-dev / web

Getting websocket error with @web/test-runner-visual-regression #2663