Launch solution for testing / demo'ing / QA'ing VA.gov

rroueche commented 5 years ago

Product Outline

https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/platform/preview/product-outline.md

Use Cases

As a call center rep, in order to advise users of VA.gov, I need to familiarize myself with the logged-in experience of VA.gov.
As a VFS sprint team, I need to demo my app to stakeholders when it's still a WIP and after it's already built (including to call centers, VA leadership, unsure if there are more...?) to show how my app works.
As a VFS sprint team, I need to QA my app myself (i.e. I shouldn't have to rely on sitting with real users in UAT for Q/A).
As a VFS sprint team or the VSP Triage team, I need to be able to reproduce issues that users are seeing.

Challenges:

VA has said "no" to getting prod users that aren't actually really people into MVI and MHV. David Bao and others have tried to push on this.
The existing massive index of staging users and what they should/shouldn't have access to is hard to use.
Dependent systems in staging are down all the time.
You need different users to test different pieces of functionality across the site. There's not really one user that has good data in place to get a good prod-like experience for all of the tools across VA.gov.

Definition of done:

rroueche commented 5 years ago

Notes from further convos:

Open to stop gaps to help unblock Call Center Reps and VFS Teams trying to demo stuff in the immediate term, but longer term solution should probably come from Ops + Lighthouse collaboration.

desireeturner commented 5 years ago

From the Booz Allen team with regard to our support of Education Services on VA.gov (Form 22-1995 & GI Bill Comparison Tool):

When we're updating functionality that is already live on Production, our process has expanded to include the following steps:

adding a prod flag to the functionality
receiving external approvals on the changes
testing and merging

Then, we repeat all of those steps again to remove the production flags after we have completed testing on staging.

If we had a preview or demo environment we could remove a lot of that overhead and avoid the Prod Flag process. And while we currently have this capability, it requires Socks access which means our clients (Education Services) are unable to access the changes.

KevinMHoffmanUSDS commented 5 years ago

Another comment from the Booz team:

I think this ticket does mostly address the same issue we raised. Our specific needs are the ability to have possibly several demo instances in a staging or equivalent environment, so that our external customers can review and being able to demo for an extended time without concern of other teams affecting the build/deploy during demo/test.

Note: several demo instances

KevinMHoffmanUSDS commented 5 years ago

Also BAH: When doing usability testing against updates to the GI Bill Comparison tool our challenge was that, because the tool is so highly interactive, creating a prototype that included the full array of functionality (with real data) was not feasible. Test participants quickly realized the functionality of the prototype was limited and, as a result, were less likely to attempt to engage with the prototype version of the tool. We saw a marked increase in engagement with all the controls of the tool when we were able to use the review instance.

jchasia-adhoc commented 4 years ago

Could we talk about how this could be a Next or Future priority?

johnpaulashenfelter commented 4 years ago

@rroueche seems like there's not a lot actionable here for fe/be tools until this is truly prioritized. This should be in Future Epics I believe.

department-of-veterans-affairs / va.gov-team