Enhance VRT Infrastructure - Constrain to Viewport (With Option to Override) & Allow Specified Scenario Runs Only

[10upsimon]

Feature Description

Enhancement 01

While working on #9375 it was discovered that elements that should be fixed to the bottom of the viewport (or other such element) - via position: fixed; bottom: 0; style implementations - are appearing as floating in the middle of nowhere in the actual VRT test bitmaps. An example of this can be seen below:

Upon assessing the viewport definitions at tests/backstop/viewports.js, specifically taking the small viewport into account...

    {
        label: 'small',
        width: 420,
        height: 580,
    },

... It was discovered that the apparent floating element was indeed ending at the 580px vertical mark, indicating that - according to the defined viewport size - it was indeed correctly positioned.

Further investigation revealed that none of the VRT test images were adhering to the viewport constraints. This is due to the default selector in Backstop being document. Therefore, the entire document as being captured.

The selectors property can be applied to scenarios to specify which selector you'd like Backstop to "capture". An obvious fix for this seemed to be the application of selectors: [ 'viewport ] to each scenario.

In theory, this would have been an acceptable approach, however some stories are more style-guide in nature, and require the full canvas of of the story to be captured during testing, and our existing implementation of Backstop and/or Puppeteer does not appear to respect this selector. Examples stories include:

• Type scales
• Buttons
• Circular progress bars
• Etc...

Digging deeper, it was clear that two key features are needing to be introduced to solve this issue:

1. The ability to constrain the story to the viewport by default
2. The ability to "break out" of this constraint for stories that require it

In order to achieve this, a few key aspects will need to be introduced, namely:

1. Introduction of an (optional) story parameter: constrainToViewport with a default value of true
2. Parsing of this property at a scenario level in tests/backstop/scenarios.js for both modern and legacy story traversals
3. Based on the above, conditionally setting the value of the selectors property for each scenario to one of the following:
   • selectors: [ '#document ], for stories that should be constrained to the viewport dimensions (default), more details on this further below
   • selectors: [ 'document' ] for stories that should not be constrained to the viewport dimensions
4. An additional decorator within .storybook/preview.js that conditionally renders a new div wrapper with ID document for viewport constraint stories. This additional element would have CSS that constrains it's dimensions to the viewport, and prevents overflow of content. This is required in order to fix the default behaviour of Backstop not adhering to the [ 'viewport' ] selector. This element should purposefully be named #document so that all existing VRT images that do continue to pass are named identical to the default [ 'document' ] selector, thus limiting the failures. If this was not the case, ALL existing VRT tests would fail.

Following the above, a story who wishes to break out of the viewport constraint would simply pass the applicable parameter:

...
decorators: [
        ( Story, { args } ) => {
                     ...
        },
    ],
        {
            parameters: {
                constrainToViewport: false,
            }
        }

Thus allowing the fill canvas to be captured as prior.

Not only will this transform the VRT captures to be true to our viewport definitions, it'll also reduce the size of the VRT reference folder and the test payload itself. Although this may not have an execution-time based impact, it's still less energy, a less wasteful operation overall.

Enhancement 02

While exploring possible solutions related to the above, it was becoming increasingly clear that having to run the entire VRT suite over and over was becoming a) time consuming and b) more prone to error should system resources become unstable (resulting in false failures, timeouts etc).

Part of this VRT enhancement sees the ability to specify a single or multiple scenario labels that should be run as part of the VRT test. This is a simple yet effective enhancement, and will greatly reduce developer overhead when needing to run and re-run VRT tests locally when working on a single or small handful of component updates.

Given the fact that story scenarios are assigned to a single constant, this can easily be achieve by:

• Using existing node features like process.argv
• Extracting string values from a logically named argument such as --labels passed to the npm test:visualtest command
• Obtaining an array of scenario labels therefrom
• Filtering all scenarios via scenarios.filter() and matching against those passed to the --labels arg

This would allow developers to run single or multiple story scenarios for purposes of VRT via a command like or similar to the following:

npm run test:visualtest -- --labels="KeyMetrics/MetricSelectionPanel"

This would be a welcome enhancement for local development.

---

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

• VRT test bitmaps generated during the execution of the test:visualtest job should be constrained to the sizes defined in tests/backstop/viewports.js unless otherwise specified via a new constrainToViewport story scenario property
  • For tests constrained to the viewport, file names should adhere to the current naming standards in order to not fail a large portion of the existing tests.
• A new decorator should be added to .storybook/preview.js that conditionally renders the required markup for the above to be satisfied, and should be the last decorator in the file. This decorator should be aptly documented as such, given the importance that it remains the last decorator in the list.
  • Said decorator should, when necessary, wrap the story in a div with id document with applicable inline styles that constrain the rendered story to the viewport width and height, with no overflow. This element ID is aptly named document to preserve file naming conventions as per the point above.
• Developers should be able to specify scenario labels that should be run in the test:visualtest job via the addition of an optional --args argument that can be passed to the job cli command, i.e npm run test:visualtest -- --labels="Some/Label".
• Existing stories that require full browser canvas captures for test purposes should be updated accordingly to utilise the newly added constrainToViewport parameter, specifying a boolean value of false. Stories that should adhere to the viewport constraints (default behaviour) should not be updated, as this will be the default outcome.
• All VRT reference images should be updated following a full run of the test:visualtest job.
• A subsequent a final run of the test:visualtest job should render no test failures.

Implementation Brief

• [ ]

Test Coverage

QA Brief

Changelog entry

[aaemnnosttv]

@10upsimon to summarize our recent discussions

• Set the default of constrainToViewport to false to keep things closer to how they are with Backstop by default and what our images contain to-date. The constrain: true case should be an edge case anyways as there's only been the one (?) so far where it's been important
• When constrain: true is something we want for VRT, we can set selectors: #document for those scenarios. While this is not ideal, it saves the effort of applying this automatically for now which we can do in a follow up as this involves a more complicated change in the babel traversal to "find" that property for relevant stories
• Scenario filtering (Enhancement 2) – this addition is unnecessary as this is already possible with the core Backstop flag --filter [filter]