Medallia Survey Tool: Release Plan

[jwoodman5]

Guidance

This Release Plan Template is intended to help ensure your new VSP product or feature is ready for launch and will deliver the desired user outcomes.

• In Phase 1, you will test your product to learn how it'll actually be used, what problems it might create, and then fix/adjust if necessary prior to going live.
• In Phase 2, you will define the launch communications plan and post-launch KPIs. You can close this ticket once you have completed Phase 2.
• Phase 3 is optional - you may use it to document post-launch results.

Fill out the Planning sections to start, then the Results sections as you complete each phase of testing.

---

Phase I: User Acceptance Testing

Test real-world user scenarios to ensure your product or feature performs as expected.

Planning:

• Desired date range or test duration: Nov. 18-20 & 23-24, 2020

• Desired number of users: 9

• The group(s) from which you'll recruit test users: VFS/VSP

• How you'll recruit test users: 3rd party company - Perigean

• How you'll give test users access to the product: Zoom meeting URL and screen share

• What UI/UX "Success" criteria will you look at before launch?

  Can users find the forms? What are their impressions of the forms? What would they use them for? What are their expectations after forms are submitted?

• How you'll measure your UI/UX "Success" criteria?

  'Feedback' button difficulty, findability, time to find 'Feedback' survey experience questions - 1. How did this interaction change your trust in Veteran Affairs? 2. How much do you trust Veteran Affairs to fulfill our country’s commitment to Veterans?

• What general "Success" criteria will you look at before launch?

DEPO is able to use the feedback information to:

• Understand overall va.gov satisfaction
• Understand va.gov satisfaction by benefit hub
• Easily report customer satisfaction and trust to varying stakeholders DEPO has VA.gov user trust & satisfaction data collected consistently and can data be easily
• Verified for accuracy
• Pulled from a single source of truth
• Compare trust and satisfaction data between VFSs, benefit hubs, and VA.gov as a whole The VA has a consistent way of collecting veteran trust and satisfaction The VA has a more open network of sharing data, including veteran trust and satisfaction data
• How you'll measure your general "Success" criteria? Percentage of VA.gov users providing feedback i. Per product CSAT i. All of VA.gov ii. Per benefits hub iii. Per product Percentage of live modernized VFSs with surveys

Results (of UI/UX research):

• Number of users: 9

• Results per your "Success" criteria: The 'Feedback' button was very difficult to find. Only 1/9 Veterans found it within 1-2 minutes

• Number of bugs identified / fixed: 1 / 1 (button findability)

• Any UX changes necessary based on the logs, user feedback, or VA challenges? Yes

  • If yes, what:

    Change 'Feedback' button color/design to make it easier for Veterans to find on VA.gov pages TBD

Phase II: Go Live!

Planning:

• Desired launch date: 1/27/21 (Pilot) - 3/3/21 (Full release)
• Implementation Guide Here for now.
• Post-launch KPI 1: % of users who submit Feedback surveys (# of surveys submitted/unique page visits)
• Post-launch KPI 2: % of surveys converted from ForeSee to Medallia
• Post-launch KPI 3: TBD

(Q1 2021) Feedback Survey Rollout:

• Begin the Feedback survey
• Start with teams ready to migrate from ForeSee to Medallia using the approved 3-question survey
• Next, add new standard Medallia Feedback surveys in sequential order for teams who have requested them
• Handle non-standard Feedback survey requests on a case-by-case basis

(Late Q1/early Q2 2021) Intercept Survey Rollout:

• Rollout custom, 4-question intercept survey with ORMDI team as constrained pilot
• (1-3 months) Pilot the standard, approved Medallia intercept survey alongside ForeSee intercept survey to analyze CSAT scoring
• (Late Q2/early Q3 2021) Once a Medallia CSAT score has been determined, release VA.gov-wide (minus exclusions)
• Catalog custom intercept survey requests and handle on a case-by-case basis

Phase 3: Post-launch Results [Optional]

1-week results:

• Number of unique users:
• Post-launch KPI 1 actual:
• Post-launch KPI 2 actual:
• Post-launch KPI 3 actual:
• Any issues with VSP handling/processing?: yes/no
  • If yes, what:
• Any UX changes necessary based on the logs, user feedback, or VA challenges? yes/no
  • If yes, what:

1-month results:

• Number of unique users:
• Post-launch KPI 1 actual:
• Post-launch KPI 2 actual:
• Post-launch KPI 3 actual:
• Any issues with VSP handling/processing?: yes/no
  • If yes, what:
• Any UX changes necessary based on the logs, user feedback, or VA challenges? yes/no
  • If yes, what:

Post-launch Questions

To be completed once you have gathered your initial set of data, as outlined above.

1. How do the KPIs you gathered compare to your pre-launch definition(s) of "success"?
2. What qualitative feedback have you gathered from users or other stakeholders, if any?
3. Which of the assumptions you listed in your product outline were/were not validated?
4. How might your product evolve now or in the future based on these results?

[Kimberley2019]

Living Doc. On hold currently until phase two is complete.

[Kimberley2019]

Phase two is currently in progress.

[jwoodman5]

We are about 30% done with 508 fixes. 2 of 3 launch-blocking 508 fixes are done. We are looking to pilot the intercept survey, which is now free of launch blocking issues and will do the same with the Feedback survey once the last launch-blocking issue is resolved (by Medallia). Still targeting "Go Live" by end of Q1.

[jwoodman5]

Releases for both the Feedback and Satisfaction surveys are successfully complete. Related work handled in other initiatives/epics/stories. Closing.