test(visual): improve stability

[giamir]

STACKS-512

Test run seems to very stable now (see here).

@dancormier When you get a second (not urgent) can you test in your machine and give feedback? Thanks

[netlify[bot]]

✅ Deploy Preview for stacks ready!

Name	Link
🔨 Latest commit	a8a980da8ba3967da9da60d5f927a06b5612528d
🔍 Latest deploy log	https://app.netlify.com/sites/stacks/deploys/65c0b908b0fb5c00085ddc98
😎 Deploy Preview	https://deploy-preview-1626--stacks.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dancormier]

@giamir I haven't given this PR a thorough review, but I wanted to let you know the results of my local run.

Shell output (run 1)

```sh Code/Stacks  STACKS-512/improve-visual-regression-tests-stability [$?] v2.0.9 ⬢ v20.10.0 🐳 v24.0.7 (desktop-linux) ➜ npm run test 05:06:44 PM > @stackoverflow/stacks@2.0.9 test > npm run test:less && npm run test:unit && npm run test:a11y && npm run test:visual > @stackoverflow/stacks@2.0.9 test:less > vitest run .less.test.ts RUN v1.2.1 /Users/dan/Code/Stacks ✓ lib/atomic/v1/color.less.test.ts (1) 828ms ✓ lib/exports/color.less.test.ts (1) 458ms ✓ lib/atomic/v1/border.less.test.ts (1) 383ms ✓ lib/atomic/color.less.test.ts (1) 504ms ✓ lib/exports/v1/constants-colors.less.test.ts (1) ✓ lib/exports/color-mixins.less.test.ts (7) 599ms ✓ lib/atomic/v1/typography.less.test.ts (1) 334ms Test Files 7 passed (7) Tests 13 passed (13) Start at 17:07:12 Duration 1.57s (transform 680ms, setup 2ms, collect 1.95s, tests 3.38s, environment 2ms, prepare 1.19s) > @stackoverflow/stacks@2.0.9 test:unit > web-test-runner --group=unit Chromium: |██████████████████████████████| 6/6 test files | 19 passed, 0 failed Firefox: |██████████████████████████████| 6/6 test files | 19 passed, 0 failed Webkit: |██████████████████████████████| 6/6 test files | 19 passed, 0 failed Finished running tests in 50.6s, all tests passed! 🎉 > @stackoverflow/stacks@2.0.9 test:a11y > web-test-runner --group=a11y Chromium: |██████████████████████████████| 35/35 test files | 1770 passed, 0 failed, 48 skipped Finished running tests in 44.4s, all tests passed! 🎉 > @stackoverflow/stacks@2.0.9 test:visual > pwsh run-test-visual.ps1 npx web-test-runner --group=visual lib/components/badge/badge.visual.test.ts: ❌ Browser tests did not start after 20000ms You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information. (failed on Chromium and Webkit) ❌ The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox) Error: The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. at Timeout._onTimeout (/app/node_modules/@web/test-runner/node_modules/@web/test-runner-core/dist/utils/async.js:14:20) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) lib/components/banner/banner.visual.test.ts: ❌ Browser tests did not start after 20000ms You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information. (failed on Chromium and Webkit) ❌ The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox) Error: The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. at Timeout._onTimeout (/app/node_modules/@web/test-runner/node_modules/@web/test-runner-core/dist/utils/async.js:14:20) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) lib/components/block-link/block-link.visual.test.ts: ❌ Browser tests did not start after 20000ms You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information. (failed on Chromium and Webkit) ❌ The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox) Error: The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. at Timeout._onTimeout (/app/node_modules/@web/test-runner/node_modules/@web/test-runner-core/dist/utils/async.js:14:20) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) lib/components/breadcrumbs/breadcrumbs.visual.test.ts: ❌ Browser tests did not start after 20000ms You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information. (failed on Chromium and Webkit) ❌ The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox) Error: The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. at Timeout._onTimeout (/app/node_modules/@web/test-runner/node_modules/@web/test-runner-core/dist/utils/async.js:14:20) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) lib/components/avatar/avatar.visual.test.ts: ❌ Browser tests did not start after 20000ms You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information. (failed on Chromium and Webkit) ❌ The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox) Error: The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. at Timeout._onTimeout (/app/node_modules/@web/test-runner/node_modules/@web/test-runner-core/dist/utils/async.js:14:20) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) lib/components/activity-indicator/activity-indicator.visual.test.ts: ❌ Browser tests did not start after 20000ms You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information. (failed on Chromium and Webkit) ❌ The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox) Error: The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. at Timeout._onTimeout (/app/node_modules/@web/test-runner/node_modules/@web/test-runner-core/dist/utils/async.js:14:20) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) lib/components/anchor/anchor.visual.test.ts: ❌ Browser tests did not start after 20000ms You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information. (failed on Chromium and Webkit) ❌ The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox) Error: The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. at Timeout._onTimeout (/app/node_modules/@web/test-runner/node_modules/@web/test-runner-core/dist/utils/async.js:14:20) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) lib/components/award-bling/award-bling.visual.test.ts: ❌ Browser tests did not start after 20000ms You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information. (failed on Chromium and Webkit) ❌ The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. (failed on Firefox) Error: The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option. at Timeout._onTimeout (/app/node_modules/@web/test-runner/node_modules/@web/test-runner-core/dist/utils/async.js:14:20) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) Chromium: |██████████████████████████████| 35/35 test files | 2008 passed, 0 failed Firefox: |██████████████████████████████| 35/35 test files | 2008 passed, 0 failed Webkit: |██████████████████████████████| 35/35 test files | 2008 passed, 0 failed Error while running tests. ```

Besides visual regression tests, everything seemed to run smoothly. For the visual tests, I had 8 reported timeouts which all looked something like this:

lib/components/badge/badge.visual.test.ts:

 ❌ Browser tests did not start after 20000ms You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information.  (failed on Chromium and Webkit)

 ❌ The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option.  (failed on Firefox)
    Error: The browser was unable to create and start a test page after 30000ms. You can increase this timeout with the browserStartTimeout option.
        at Timeout._onTimeout (/app/node_modules/@web/test-runner/node_modules/@web/test-runner-core/dist/utils/async.js:14:20)
        at listOnTimeout (node:internal/timers:569:17)
        at process.processTimers (node:internal/timers:512:7)

After this first run, I ran it again and had a similar outcome. If you'd like I'm happy to help you debug this in whatever way might help @giamir

[giamir]

@dancormier Running visual regression tests requires a significant amount of resources (since 3 browsers are running in parallel multiple tests). The good thing is that all the failures you get are timeouts. This in my opinion means that you have not allocated enough resources to the docker engine in your machine (or more unlikely that you need a more powerful machine). 😀

I will start by tweaking the docker settings. Here are mine as a reference:

Let me know if this helps otherwise we might want to tune down a bit the parallelism settings in WTR (ideally not though because it will inevitably slow the test run).

[giamir]

@dancormier Given that running individual tests in parallel did not yield much benefits and created more flakiness I opted to simplify the setup and keep only the parts that are helping to keep the suite stable. In particular I removed the semaphore and the bespoke visual test framework and I kept the retry mechanism to guarantee stability. I also set the number of concurrent browsers to 3 because it speeds up tests also in CI. If you have issues in your machine with that setting I am happy to set the number only in the dedicated wtr-ci config file.

Despite the 77 file changes this PR doesn't change a lot - most of those changes are references to the new test-utils files. The file that deserves a proper review is visual-test-utils.ts. The rest of the stuff is just cosmetic.

Thank you in advance for the review.