FE Testing Standards - Deliver a comprehensive and cohesive suite of testing tools\processes

[brandonrapp]

Overview

We want the most comprehensive testing possible with the least amount of configuration required. We have some tools in place now, but we do not always agree on a definitions for our testing methodologies (eg. e2e Testing). We need to audit our existing solutions and do discovery on new solutions so we can decide on what tools\processes we are going to utilize for our standards.

VSP Q1 + Q2 2020 OKRs

O5: All customers comply with the Platform's standards.

Pain Points:

VFS Teams - Documentation outlines platform expectations and standards for testing

Goals

• Provide a clear explanation of the standard platform process for testing (FE)
• Definitions of our methodologies and agreed on testing terminology
  • Methodologies: Unit, Functional, Integration, Accessibility
  • Define - What is e2e testing for the platform
• Clear documentation for each testing element organized under the correct methodologies
• A narrative that explains how they are used in conjunction for comprehensive testing
• A way to measure how compliant teams are towards our standard process

Potential tasks

• Discovery work around our current tools
• Talk to our users about testing requirements
• Identify and agree on our standard methodologies
• Identify a way to measure VFS Team compliance towards these methodologies
• Gather feedback and iterate on our standards
• Perform a gap analysis around our current tools and our standards
• Identify tools\processes to fill any gaps and begin discovery
• Design Documents for each new tool\process
• Create and circulate an implementation\roll out plan
• Create or update our documentation to align with our standards

Testing Tools to audit\evaluate

• Unit Testing:
• Functional Testing:
• Integration Testing:
• Accessibility Testing:

Questions

• What about Visual Regression Testing?
• What is our standard definition of e2e testing?

[brandonrapp]

Here is the document we can use for tomorrow's scope and planning discussion. https://docs.google.com/document/d/1Isn6lb7U45n3l0TxPHyG5zIqhNRIcsqG5ZvmyG1IL3I/edit?usp=sharing

[dbru]

Link to discovery interview notes

https://drive.google.com/drive/u/0/folders/1cu4Mx6jJrNLMh7Fwe2ykVtRjZD7UBbhG

[brandonrapp]

Full Notes from our FE Testing Tool Discussion is here - https://docs.google.com/document/d/1Isn6lb7U45n3l0TxPHyG5zIqhNRIcsqG5ZvmyG1IL3I/edit

Gaps Identified:

• Cross Browser testing - no automated way right now
  • Make a decision on this
  • Doesn’t need to be perfect but we need something
• Cypress vs TestCafe
  • Auto Form Tester - reduce the amount of tests you have to write
  • Would not help every team (VAOS), but Cypress could be better
  • Would allow us to potentially include more automated axe-checks
  • Need to figure out the narrative about why switching is better for the platform
  • Auto Form Tester - Not compatible with Nightwatch
  • Cross-browser Testing
  • Nightwatch with Sauce Labs - is clunky
• Integration testing - Need a way to test against vets-api
  • Need to figure out a direction WITH BE
• Accessibility Testing - Teams are interested in testing via browser plugin
  • Is there another framework that would help us with this
• Documentation needs to be looked at
  • What kind of coverage do teams need to put into their app, but that is tricky
  • Testing Best Practices
  • Better to show via examples - Looking at an example is better representative for how.
• Mock API Documentation is not directly

Follow-ups:

• Talk with Trevor about his testing thoughts
• Auto Form Tester with Cypress
  • Get it to run on HCA
  • Faster Testing Performance (speed)
  • Configuration Burden (# of tests)
  • Amount of Coverage Provided (rougher)
  • Number of breakages because of Nightwatch (can hold up PR)
• Switching frameworks documentation
  • Migration Support
  • Support Nightwatch for older tests?
  • Documentation Updates
  • Alignment with QA about our switch
• Integration Testing
  • Discussions with JP - Brandon to setup
  • QASP Metric - Number of top ten VA.gov user experiences, based on unique transactions per month, that have a corresponding automated end-to-end browser test that exercises multiple pieces of functionality including both FE and BE.

[rianfowler]

rollout plan

[rianfowler]

Tracking VAOS feedback here: https://github.com/department-of-veterans-affairs/va.gov-team/issues/10048

[U-DON]

Functional Testing

Replaced some existing tests in HCA and all-claims using the Cypress form tester.

• HCA Cypress Form Test
• Disability Benefits (All Claims) Cypress Form Test

[meganhkelley]

Meeting 7/30 with @U-DON @rianfowler @1Copenut re: ESLint rule enforcing Nightwatch axe checks

Decisions

• We need a long term strategy for axe tests and how to enforce them
• Our short term strategy is to
  • push people to use the form tester
  • add an ESLint rule that applies to Cypress
    • With a warning message that links to some documentation that tells people that they can disregard this if they used the form tester and if they want to disregard the test permanently for their form they can get Trevor’s approval to do so
• Note that if/when we deprecate Nightwatch entirely, we’ll need to rewrite some helper functions for keyboard e2e tests, specifically for Education Feedback Tool and GI Bill Comparison Tool (maybe Facility Locator?)

[meganhkelley]

Cypress post-release backlog has been consolidated into this sub-epic: #12206