[HOLD] chore: reduce number of VRT combinations

rubencarvalho commented 3 weeks ago

Description

This PR reduces the number of Visual Regression Tests (VRTs) to simplify our test suite while maintaining full coverage. Previously, we had 24 VRTs across various combinations of themes, colors, scales, and directions. This setup introduces two specific test combinations, for each system we will capture:

Light theme, Medium scale, LTR
Dark theme, Large scale, RTL

These combinations ensure that all individual parameters are still covered while reducing the overall number of tests, making the VRT process faster, less prone to flakiness, and easier to maintain.

Motivation and context

Reducing the number of VRTs helps streamline our testing workflow by eliminating redundancy, minimizing test time, and reducing potential flakiness. The proposed combinations covers each parameter option (theme, color, scale, direction) without unnecessary duplication, ensuring that we still test each setting.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[x] Chore (minor updates related to the tooling or maintenance of the repository, does not impact compiled assets)

Checklist

[x] I have signed the Adobe Open Source CLA.
[x] My code follows the code style of this project.
[x] If my change required a change to the documentation, I have updated the documentation in this pull request.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.
[x] I have reviewed at the Accessibility Practices for this feature, see: Aria Practices

Best practices

This repository uses conventional commit syntax for each commit message; note that the GitHub UI does not use this by default so be cautious when accepting suggested changes. Avoid the "Update branch" button on the pull request and opt instead for rebasing your branch against main.

github-actions[bot] commented 3 weeks ago

Branch preview

github-actions[bot] commented 3 weeks ago

Tachometer results

Currently, no packages are changed by this PR...

coveralls commented 3 weeks ago

Pull Request Test Coverage Report for Build 11781709628

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.002%) to 98.195%

Totals
Change from base Build 11780949750:	-0.002%
Covered Lines:	32328
Relevant Lines:	32745

💛 - Coveralls

github-actions[bot] commented 3 weeks ago

Lighthouse scores

Category	Latest (report)	Main (report)	Branch (report)
Performance	0.99	0.99	0.99
Accessibility	1	1	1
Best Practices	1	1	1
SEO	1	0.92	0.92
PWA	1	1	1

What is this?

[Lighthouse](https://github.com/GoogleChrome/lighthouse) scores comparing the documentation site built from the PR ("Branch") to that of the production documentation site ("Latest") and the build currently on main ("Main"). Higher scores are better, but *note that the SEO scores on Netlify URLs are artifically constrained to 0.92.*

Transfer Size

Category	Latest	Main	Branch
Total	250.801 kB	236.795 kB 🏆	236.947 kB
Scripts	61.06 kB	54.345 kB	54.33 kB 🏆
Stylesheet	53.732 kB	48.012 kB 🏆	48.12 kB
Document	6.222 kB	5.469 kB 🏆	5.479 kB
Font	126.773 kB	126.614 kB 🏆	126.642 kB

Request Count

Category	Latest	Main	Branch
Total	52	52	52
Scripts	41	41	41
Stylesheet	5	5	5
Document	1	1	1
Font	2	2	2

adobe / spectrum-web-components