Research and Implement Visual Regression Testing in CI/CD

[pjhill]

Problem Statement

VFS Teams do not currently have a means for verifying that changes are not indirectly causing regressions elsewhere in the platform. VSP personnel do not have any means of continuously scanning for visual regressions on the platform as changes are integrated.

How might we verify that newly integrated changes are not inadvertently causing regressions on the platform? How might we collect information about visual regressions into a report or dashboard?

Hypothesis or Bet

Given the ability to observe the impact to regressions on the platform immediately upon integrating their changes, VFS teams can fix regressions with context and precision. A tighter feedback loop for visual regressions will empower teams to resolve issues more quickly.

We will know we're done when... ("Definition of Done")

• [x] Visual regression testing tool has been selected.
• [ ] Visual regression testing tool has been procured.
• [ ] Visual regression testing tool has been configured and installed on the platform.
• [ ] Visual regression testing tool use and process has been documented and published for use by VFS teams.

Known Blockers/Dependencies

No known blockers

Projected Launch Date

End of Q2

Launch Checklist

Guidance (delete before posting)

This checklist is intended to be used to help answer, "is my VSP initiative ready for launch?". All of the items in this checklist should be completed, with artifacts linked---or have a brief explanation of why they've been skipped---before launching a given VSP initiative. All links or explanations can be provided in Required Artifacts sections. The items that can be skipped are marked as such.

Keep in mind the distinction between Product and Initiative --- each Product needs specific supporting documentation, but Initiatives to improve existing Products should reuse existing documentation for that Product. VSP Product Terminology for details.

Is this service / tool / feature...

... tested?

• [ ] Usability test (TODO: link) has been performed, to validate that new changes enable users to do what was intended and that these changes don't worsen quality elsewhere. If usability test isn't relevant for this change, document the reason for skipping it.
  • [ ] ... and issues discovered in usability testing have been addressed.
  • Note on skipping: metrics that show the impact of before/after can be a substitute for usability testing.
• [ ] End-to-end manual QA or UAT is complete, to validate there are no high-severity issues before launching
• [ ] (if applicable) New functionality has thorough, automated tests running in CI/CD

... documented?

• [ ] New documentation is written pursuant to our documentation style guide
• [ ] Product is included in the List of VSP Products
  • List the existing product that this initiative fits within, or add a new product to this list.
• [ ] Internal-facing: there's a Product Outline checked into products/platform/PRODUCT_NAME/
  • Note: the Product Directory Name should match 1:1 with the List of VSP Products
• [ ] External-facing: a VFS-facing README exists for this product/feature tool
  • [ ] ... and should be located at platform/PRODUCT_NAME/README.md
• [ ] External-facing: a User Guide exists for this product/feature/tool, and is updated for changes from this initiative
  • [ ] ... and should be linked from the VFS-facing README for your product
  • [ ] ... and should be located within platform/PRODUCT_NAME/, unless you already have another location for it
  • [ ] (if applicable)... and post to #vsp-content-ia about whether this should be added to the Documentation homepage
• [ ] (if applicable) Post to #vsp-service-design for external communication about this change (e.g. VSP Newsletter, customer-facing meetings)

... measurable

• [ ] (if applicable) This change has clearly-defined success metrics, with instrumentation of those analytics where possible, or a reason documented for skipping it.
  • For help, see: Analytics team
• [ ] This change has an accompanying VSP Initiative Release Plan.

Required Artifacts

Documentation

• PRODUCT_NAME: directory name used for your product documentation
• Product Outline: link to Product Outline
• README: link to VFS-facing README for your product
• User Guide: link to User Guide

Testing

• Usability test: link to GitHub issue, or provide reason for skipping
• Manual QA: link to GitHub issue or documented results
• Automated tests: link to tests, or "N/A"

Measurement

• Success metrics: link to where success metrics are measured, or where they're defined (Product Outline is OK), or provide reason for skipping
• Release plan: link to Release Plan ticket

TODOs

• [ ] Convert this issue to an epic
• [ ] Add your team's label to this epic

[humancompanion-usds]

Just a note, should we eventually pick this initiative up, since this was originally filed the Design System Team has implemented visual regression testing using Chromatic. In fact, we used it not too long ago to detect a change we picked up from USWDS that impacted a component of ours which was not documented in their release notes. We did do an evaluation of different solutions.

I don't know that our solution scales, and there is cost involved, but I do agree with the premise of this initiative. When the Design System Team picks up the injected header and footer work I have made it a prerequisite that we have some similar visual regression testing for a sampling of sitewide pages so that we can have some sense that any change there is not causing an immediately user visible problem.