Improve Preview Environments for VA.gov Products

[andreahewitt-odd]

Problem Statement

The current approach to review instances has limitations that cause frustration for VFS teams and VA stakeholders in their efforts to test and preview their products. Examples of those limitations include the required use of SOCKS proxy which involves a level of technical fluency in its setup and use that most non-developers don’t possess, it also limits the tooling that can be used to conduct QA in review instances. Other limitations include that review instances are available for a limited window of time, they cannot be used to test on real mobile devices, and the developer workflow for preview of products that require combinations of multiple repositories is difficult. Additionally, review instances can only point to preprod/Staging environment so they cannot easily be configured to test against lower environments or synthetic data. Finally, better documentation and training on how to use review instances properly and fully would improve the experience.

How might we make it easier for VFS teams and VA stakeholders to preview their features before it makes it to production (especially as we move to a model of continuous deployment)?

User Impact

VFS Teams without SOCKS or the technical knowledge to use SOCKS: Can't use review instances VA Stakeholders without SOCKS or the technical knowledge to use SOCKS: Can't use review instances VFS engineers: Workflow for creation of Review Instances using branches from multiple repos is difficult, can't test with RI on non-GFE devices such as mobile, difficult to match Production data leads to a lack of confidence in the code

Where was this problem reported?

This problem has been reported everywhere (support requests, research, satisfaction survey). It's a common theme in the satisfaction survey quarter to quarter.

What do we not know about the problem space?

• How best to handle environment creation in the context of multiple repositories. Should they be created by PR within individual repo (such as vets-website and vets-api) or should there be a centralized repo that creates Preview Environments?
• Is there a network path that allows off-network (non-SOCKS) access while also connecting to preprod external integrations?
• If not, is there a minimal functionality that needs to be mocked to allow testing use cases?
• What authN/authZ systems are available and accessible to majority of VFS team members, POs, and stakeholders?
• What tools and application hosting systems (Codespaces, Tugboat, EKS, etc) are available and what limitations do they have?

What (if any) research or discovery has been done?

Satisfaction survey quarter to quarter VFS team's testing experience study - findings Q3 2022 VFS team's testing experience study - synthesis board Q3 2022

Important Project Links

• Project board
• Confluence space

What is the acceptance criteria?

• [ ] Full stack necessary to run VA.gov exists within environment (vets-website, vets-api, content-build)
• [ ] User access to environment does not require SOCKS
• [ ] User access authentication and authorization control mechanism exists
• [ ] Partial feature parity with legacy Review Instances
  • [ ] Default branch selection can be overridden
  • [ ] Preview Environments are created automatically by Pull Requests
  • [ ] Developer terminal access to environment is available
  • [ ] Environments should be paused when not in use, and destroyed when PRs are merged
  • [ ] External integrations point to Staging/Preprod by default*
  • [ ] Login is functional using Staging test users*

Technical Requirements

• [ ] Does not rely on legacy BRD infrastructure (Jenkins)

Not included in the AC

• [ ] Provisioning of test data or users in external systems
• [ ] Creation of external system mocks. *

*Note: may be affected by "Is there a network path that allows off-network (non-SOCKS) access while also connecting to preprod external integrations?"

How should we measure success?

• Fewer instances of Platform's testing ecosystem as a negative theme on the customer satisfaction survey
• Decline in support requests around review instances (need to start capturing baseline early) @JoeTice @pjhill
• Increase in usage of preview environments (need to start capturing baseline early) @JoeTice @pjhill
• Error rate and reliability of preview environments (actual equation for this will be reliant on implementation details)
• Take the pieces of the QA health survey that are relevant to review instances and compare to the recently completed survey

TODOs

• [x] Convert this issue to an epic
• [x] Assign project team name
• [x] Identify this project's OCTO and Vendor key POCs
• [x] Identify specific VFS teams to be part of this project's acceptance
• [ ] Identify project target end date
• [x] Conduct project kickoff meeting
• [ ] Develop and implement project communication plan

[mchelen-gov]

Let's scope this to creation of preview environments. Preview environments should allow:

• Selected versions of codebases based on branches (i.e. feature/xyz from vets-website + feature/xyz from vets-api)
• Configuration of environment settings such as environment variables
• Mocks for external services and data can be configured
• Ability to share access with non-developers (PM, designers, stakeholders, etc)
• Underlying infrastructure should not be hosted on legacy BRD

[JoeTice]

A few questions...

• The problem statement mentions 'there has been concern raised and incidents occurring due to environment promotion and the ability to reliably test' - can you expand on the concerns and incidents that are being referred to here?
• Are there other specific 'areas of focus' that stand out to you as areas that could or should be considered as areas to improve, either in the near term or long term?
• In 'What is the acceptance criteria?' - Is access to previewing code in review instances limited in any way?
• In 'Not included in the AC' - Can you expand on what you mean on 'promotion of code throughout requirements'? And on why you don't think tht should be part of the acceptance criteria for this?
• In the additional comments from @mchelen-gov
  • The comment calls out vets-website & vets-api - do we want to call out selected versions from other repositories?
  • Is there a reason that you want to scope this only to the creation of preview environments?
  • Can you provide some specific examples or use cases around the configuration of environment settings?

[andreahewitt-odd]

Does this only involve manual testing or automated testing too? Does this also include tooling to make automated tests? Potentially both

[andreahewitt-odd]

Are we conflating QA with review?

[andreahewitt-odd]

VFS teams aren't comfortable doing testing in a lower environment because they don't have the same data in dev and review instances.

[andreahewitt-odd]

We need to change the context to reviewing and testing in review instances -- what do we need to do for that piece to be effective. Getting data and information that is closely aligned to prod, easy access for anyone on the team.

[mchelen-gov]

How well do we understand the problem?

• Question: what is mechanism / UX for environment creation

[andreahewitt-odd]

QA health survey -- success metrics could be a portion of that so we can compare.

[andreahewitt-odd]

Change to: Increase in usage of review instances

[andreahewitt-odd]

Measurements of success might be affected by implementation details.

[andreahewitt-odd]

track usage in staging review - how often can we use these revised review instances

[andreahewitt-odd]

General bugginess of spinning up review instances -- will be dependent on implementation details.

[mchelen-gov]

^ success metric: error rate and reliability of preview environments

[mchelen-gov]

• Configuration of environment settings such as environment variables For MVP, can be accomplished by allowing developer terminal access.

• Mocks for external services and data can be configured For MVP, can be accomplished with existing developer tooling if we are able to connect to Staging/Preprod external services

[mchelen-gov]

@JoeTice

A few questions...

• The problem statement mentions 'there has been concern raised and incidents occurring due to environment promotion and the ability to reliably test' - can you expand on the concerns and incidents that are being referred to here?

The example I am aware of is code being deployed to Staging for testing purposes, which was inadvertently released to Production

• Are there other specific 'areas of focus' that stand out to you as areas that could or should be considered as areas to improve, either in the near term or long term?

Next steps for this project might be to build out more ready to use mocks for external services.

• In 'What is the acceptance criteria?' - Is access to previewing code in review instances limited in any way?

Updated AC.

• In 'Not included in the AC' - Can you expand on what you mean on 'promotion of code throughout requirements'? And on why you don't think tht should be part of the acceptance criteria for this?

I'm not clear on this, so it has been removed from AC.

• In the additional comments from @mchelen-gov

  • The comment calls out vets-website & vets-api - do we want to call out selected versions from other repositories?

This should include all repos necessary to build a working VA.gov, likely that will involve content-build and some others.

• Is there a reason that you want to scope this only to the creation of preview environments?

Yes, because this specific piece of tooling is the farthest "left" in the development and QA process.

• Can you provide some specific examples or use cases around the configuration of environment settings?

No, so I changed the AC to allow developer terminal access.

[mchelen-gov]

@andreahewitt-odd

they cannot preview products that require combinations of multiple repositories.

This can be done, however it is a more complex workflow: https://depo-platform-documentation.scrollhelp.site/developer-docs/using-review-instances-to-preview-changes#Usingreviewinstancestopreviewchanges-Manualcreation

Additionally, review instances don’t update when pushes are made to PRs and rely on mocked data/services so are not as useful as they could be.

This is incorrect, RI are redeployed on pushes and are pointed to Staging/Preprod environments per docs.

Updated issue to reflect this.

[andreahewitt-odd]

@JoeTice @pjhill See Chelen's comments per your information around the problem statement ^

[annekerr49]

@pjhill please can you give us a status update on this item?

[pjhill]

@pjhill please can you give us a status update on this item?

@annekerr49 -- This project is currently in the discovery phase. Currently, the team is addressing tickets in the Discovery Epic. Individual ticket statuses within that epic are available here.

[JoeTice]

[JoeTice]

Link to a documented plan in mural developed in Nov of 2022 that lays out the planned scope of MVP - mural link

[JoeTice]

Team of Teams Slide: 4/11/23

[JoeTice]

Update: 4/18/23

[JoeTice]

[JoeTice]

[JoeTice]

[JoeTice]

[JoeTice]

[JoeTice]

[annekerr49]

No match for roadmap-DMC

[JoeTice]

[JoeTice]

[JoeTice]

[JoeTice]

[EricaRobbins]

@JoeTice / @andreahewitt-odd - did this ship yet? If not, do you have a new Target Delivery Date?

[JoeTice]

Not yet, our next target milestone is April 2024, which will be the version that is made available for Usability Testing. The target delivery date will be based on what is surfaced in testing.

[AparnaNittalaUSDS]

Updated the target delivery date to Nov 30, 2024 based on the Milestone 4 (final testing for publicly accessible preview environment).

[annekerr49]

With Aparna leaving are we still on track to complete this effort in November?