Opportunities and Priorities for Comprehensive Test Plan

[gracekretschmer-metrostar]

User Story or Problem Statement

The CMS team needs to determine the priorities for filling in the testing gaps for critical pathways within CMS and how to implement the comprehensive test plan to move from current state of testing to ideal state of testing.

Description or Additional Context

The following opportunities have been identified by @keisterj-oddball:

• Collaborate with CMS and sitewide teams to create test scripts that missing for critical pathways.
• Separate test runs to make improve the performance of test scripts.
• Expand logging for test scripts.

Reference Links

• https://vfs.atlassian.net/wiki/spaces/~712020eee99537517b4d9594003707b252e24b/pages/3079766076/CMS+Test+Plan+Discovery
• https://github.com/department-of-veterans-affairs/va.gov-cms/pull/16410
• https://github.com/department-of-veterans-affairs/va.gov-cms/issues/19002
• https://github.com/department-of-veterans-affairs/va.gov-cms/issues/19004
• https://github.com/department-of-veterans-affairs/va.gov-cms/issues/19006

Steps for Implementation

• Document the opportunities for filling the gaps with the comprehensive test plan.
• Meet with CMS PO, PM, and tech lead to determine priorities.
• Create a plan of action in GitHub to implement opportunities that will be prioritized into future sprints.

Acceptance Criteria

• [ ] Determine the priorities for implementing the opportunities.
• [ ] Determine sequencing for the implementing the opportunities.
• [ ] Determine Jon's role versus the developer's role in test creation.

[keisterj-oddball]

PRIORITY 1: Increase critical path test coverage

• Currently, only 12 of 53 content types are covered by E2E tests proving their ability to be created and published by properly provisioned users.

Pros:

• Increasing critical path test coverage will help ensure the most important functionality in the CMS works as intended after each new change is introduced.

• Reduces the need for manual testing in cases requiring full regression testing efforts.

Considerations:

• Adding 41 tests to the "unsegmented" test suite may significantly impact the duration of test runs (impacts developer speed and build pipelines).

• Requires prioritization of content types to be automated.

• Likely requires Sitewide involvement for test creation.

PRIORITY 2: Separate existing test suite into "critical path" and "regression" test runs

Pros:

• Prioritizing the "critical path" test run could speed up the build pipeline.

• Regression could still be run on a regular schedule outside of the CMS pipeline providing frequent feedback on new changes.

• Developers could optionally run full regression or critical path in PR environments based on the risk of changes being introduced.

Considerations:

• This would require some input from Sitewide about where existing tests belong.

• Running only "critical path" tests in CI without increasing critical path coverage reduces the amount of test coverage for new changes in the short term, although the value provided by some of the existing tests could be debated.

• Could likely be completed internally by the CMS team.

PRIORITY 3: Persist test results outside of GH and Jenkins

Pros:

• This will assist in the assessment of test result trends and identification of flaky tests.

• This will enable the CMS team to identify common types of test failures and develop run books on how to address them.

Considerations:

• We want to be thoughtful on which test runs to persist results for, as well as retention periods due to storage constraints.

[keisterj-oddball]

I will create and link a separate ticket for each priority listed above before closing this ticket.

[keisterj-oddball]

19002, #19004, and #19006 linked above in Reference Links.

[gracekretschmer-metrostar]

Note to self: remove from sprint 18.