Pact: Release Plan - Githubissues

Phase I: User Acceptance Testing

Test real-world user scenarios to ensure your product or feature performs as expected.

Desired date range or test duration: September 2020
Desired number of users: 4 (2 developer pairs from 2 VFS teams)
The group(s) from which you'll recruit test users: VFS
How you'll recruit test users: Slack announcements and Team of Teams
How you'll give test users access to the product: product is available for use for all teams, just not required yet
What "Success" criteria will you look at before launch? We will know we're ready for full rollout when ...
- We've updated documentation to address all questions that arose in moderated testing
- Qualitative feedback on PACT
How you'll measure your "Success" criteria?
- Conduct moderated usability sessions with VFS teams to obtain feedback, address questions, and track necessary resulting updates to documentation and implementation

Number of users: 4 (2 developer pairs on 2 VFS teams)
Results per your "Success" criteria:
- VSA Facilities https://github.com/department-of-veterans-affairs/va.gov-team/issues/13393
- VSA Auth Experience https://github.com/department-of-veterans-affairs/va.gov-team/issues/13336
Number of bugs identified / fixed: see tickets above
Any UX changes necessary based on the logs, user feedback, or VA challenges? yes
- If yes, what: see tickets above, as well as #13594

Desired launch date: April 1st, 2021
Post-launch KPI 1: Number of incidents/quarter requiring postmortems related to vets-api integrations
Post-launch KPI 2: Number of CDCTs (Pacts) in broker
Post-launch KPI 3: % of Pact tests that are passing
Post-launch KPI 4: % of VFS teams who self-report using Pact for all new integrations with vets-api
Post-launch KPI 5: Qualitative feedback from VFS teams

Post-launch KPI 1 actual: Number of incidents/quarter requiring postmortems related to vets-api integrations
Post-launch KPI 2 actual: Number of CDCTs (Pacts) in broker
Post-launch KPI 3 actual: % of Pact tests that are passing
Post-launch KPI 4 actual: % of VFS teams who self-report using Pact for all new integrations with vets-api
Post-launch KPI 5 actual: Qualitative feedback from VFS teams
Any issues with VSP handling/processing?: yes/no
- If yes, what:
Any documentation changes necessary based on the logs, user feedback, or VA challenges? yes/no
- If yes, what:

Post-launch KPI 1 actual: Number of incidents/quarter requiring postmortems related to vets-api integrations
Post-launch KPI 2 actual: Number of CDCTs (Pacts) in broker
Post-launch KPI 3 actual: % of Pact tests that are passing
Post-launch KPI 4 actual: % of VFS teams who self-report using Pact for all new integrations with vets-api
Post-launch KPI 5 actual: Qualitative feedback from VFS teams
Any issues with VSP handling/processing?: yes/no
- If yes, what:
Any documentation changes necessary based on the logs, user feedback, or VA challenges? yes/no
- If yes, what:

To be completed once you have gathered your initial set of data, as outlined above.

How do the KPIs you gathered compare to your pre-launch definition(s) of "success"?
What qualitative feedback have you gathered from users or other stakeholders, if any?
Which of the assumptions you listed in your product outline were/were not validated?
How might your product evolve now or in the future based on these results?