Measure In-App Veteran Effort & Gather In-App Feedback

[kellylein]

Project Status Overview

Project Objective:

Improve our understanding of how Veterans feel about our app and get feedback about ways it could be improved

Monthly Update

Updates

|Month| Project Health | Progress & Key Accomplishments| Issues/Risks/Blockers| |-------| ---------|--------|---------| |Feb '24| Green | Product brief started which will define the specific score to track and feedback mechanism. Once reviewed with POs, implementation will start, ideally in late March. | | |Mar '24| Green | Product brief draft has been completed by Ameet and Liz and has been reviewed by multiple Global team members. Next this product brief and plan will be explored with POs. After PO review, implementation will start likely in late April or early May after the home screen has been released. In the meantime, Liz and Ameet have met with the sitewide facilities PO on Medallia and will be connecting with other folks knowledgeable on Medallia to further plan implementation | | |Apr '24| Green | Presented plan to POs. In May, the focuses will be on three main tasks. 1. Create survey questions. 2. Technical implementation of Medallia into the mobile app. 3. Designing flow, logic, and UI for surfacing survey questions to Veterans. It is unlikely that all, if any, of these items will be complete in May. | | |5/21/24| On track| Adding Medallia to the app and defining the survey question display logic are both in-progress |June '24| On track| Adding Medallia to the mobile app was paused, due to focus on the home screen, and will be resumed in the second half of June| |6/26/24| On track| Adding Medallia to the mobile app is in progress|

Detailed Project Scope

Problem Statement

• There is no clear mechanism for veterans to submit feedback–it is entirely on the veterans to determine their method which include but are not limited to:
  • Leaving a review in the Apple App Store/Google Play
  • Posting in the one of several VA-related Subreddits
  • Reaching out to general VA support via phone or email and hoping the feedback is passed along
  • Posting on VA news/blog posts
    • For example, comments from frustrated veterans on the news announcement of Rx Refill going wide release
• All current methods are all just going into a void with no clear line to even being reviewed, let alone yield results/changes desired
• The lack of a clear mechanism for feedback decreases the likelihood of veterans bothering to share their experiences which in turn reduces our capacity to improve serving them
• Both research sessions and feedback we have been passed from the VA highlight how confusing “VA mobile app” terminology is: many “mobile app users” actually use other VA apps, are referring to text messaging utilities VA offers, or even just are using VA.gov or MHV via their phone’s mobile browser

Assumptions

• Assuming that veterans are willing to give us actionable feedback as opposed to simply “venting frustrations” feedback
  • We are confident that veterans hold strong opinions about their VA interactions (including with our app) and will be receptive to providing us helpful guidance if we offer an avenue that gives them confidence we are listening
    • We strongly believe that veterans are selfless to a degree that while fixing their problems is helpful, they also care that it is fixed for other veterans poised to follow in their steps and want it to be a better experience for them
  • Not part of In-App Veteran Recruitment Project

Risks

• Open us up to receiving very unfriendly feedback that is not helpful/is offensive
  • Attempt to mitigate via copy reminding it’s going to humans on the other side
  • Potentially have the ready ability to delete unhelpful feedback/file it away to where we won’t see it without looking
• We can’t ensure that veterans won’t send us PII or other sensitive data
  • Attempt to mitigate via copy reminding them not to
  • Ensure our backend receiving the data is limited to authorized access by invite-only means controlled by the team/VA
• Have an adequate resources to monitor the intake and prioritize the feedback
• Veterans constantly submitting the same feedback until the issue is fixed or completed
• Possibility this is seen as a survey which VA has its own rules / regulations around

Definition of Done

Following needs to be true

|Item | Completed | |-------| ---------| | | | | | |

Project Scope

Items in and out of scope

|In Scope | Out of Scope| |-------| ---------| | | | | | |

Important Project Links

Links

- [Github](url) - [Product Brief](url) - [Sprint 0](https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/va-mobile-app/features/Submit%20Feedback/submitFeedback.md) - [Product Canvas](url) - Design - [Discovery](url) - [Mockups](url) - [Final Design](url) - [Figma File](url) - Research - [Document 1](url) - [Document 2 ](url) -Analytics - [Document 1](url)

Team Members

Contact List

- **OCTO Product Leads:** Rachel Han and Ryan Thurlwell - **PM:** Rachel Han - **Ad Hoc PM:** Meko Hong - **Engineering:** Jayson Perkins and Jon Bindbeutel - **Design:** Lauren Russell - **External Team 1 & Role:** - **External Team 1 & Role:**

OKR and Analytic Metrics

Metrics

|What we are measuring | Why | Event Name |-------| ---------| ----- | | | | | | | | |

Checklist

Toggle Product Checklist

**Project Kickoff & Requirements Gathering** - [ ] Project Kickoff session - [ ] Initiative Scope finalized - [ ] Link Product Brief - [Sprint 0 ](https://docs.google.com/document/d/1WHDJ1iCt9J-acLx8_DSL4tkRKoVYqJGaRUqXuFf7ceg/edit# ) - [ ] Link to Implementation Strategy - Implementation Plan - UAT Plan - Release Plan - Metrics Identified (what metrics are we capturing, how do they tie into stakeholder OKRs, etc) **Pre-Production Readiness** - [ ] Associate Readiness - [ ] Update product guide - [ ] Coordinate with contact center support teams - [ ] What’s New Content - [ ] App Store/In App messaging (if applicable) - [ ] Coordinate with Release coordinator - [ ] App store Content changes required? Y/N - [ ] App store images needed? Y/N - [ ] QA Signed off - [ ] UAT complete - [ ] Obtain Go/No Go decision **Feature Go Live** - [ ] Ensure Waygate/Feature toggle release ticket submitted w/engineering & QA - **Template: Turn on Waygate/Turn off feature flag** - [ ] Coordinate with content & release coordinator if what’s new in app is required

Toggle Team Collaboration Checklist

- [ ] Design Intent & Feasibility discussions - [X] Research needed? No - [ ] UAT Needed? Yes/No - [ ] Ticket creation - [ ] Demo feature (mid check, end of development)

Toggle Design/UX Checklist

- [ ] Review problem/opportunity statement - [ ] Add questions and assumptions to product brief - [ ] Add research links to product brief - [ ] Document high-level research/design plans - [ ] Create lo/hi fi wireframes & user flows if applicable - [ ] Add links to Epic & applicable GitHub folders - [ ] Socialize designs - [ ] Sign off received - [ ] Complete Design QA - [ ] Does it require component review? - [ ] Accessibility review needed?

Toggle Content Checklist

- [ ] Review product brief and surface any questions, assumptions & risks - [ ] Participate in project discovery and kickoff activities/ceremonies - [ ] Complete comparative analysis/content research activities for net-new content - [ ] Review current content and determine where app may need to differ and potential improvements to share with web - [ ] Review past VA research and decisions documentation - [ ] Support Research & UX Design in research sessions and synthesis - [ ] Make content recommendations for lo-/hi-fi wireframes in collaboration with UX Design - [ ] Collaborate with Sitewide Content team for alignment, improvements, and sign-off - [ ] Participate in FE hand off - [ ] Complete content QA - [ ] Write copy for What's New In App - [ ] Write copy for app stores' What's New sections. - [ ] Write copy for updated screenshots in app stores

Toggle Engineering Checklist

- [ ] Are UI designs/ specs available/ready? If yes, they should be attached to attach specs. (We are assuming the design specs have been approved) - [ ] Are there any deadlines that need to be met? - [ ] What is the roll out plan? (Do we need a feature toggle? is this going to be available to every user immediately or do we want to roll out to a few users first) - [ ] Does this project require api integration? if yes, what api endpoint are we integrating with and what is the expected response body? - [ ] Do we want to capture analytics for this projects (i.e. update GA, add logs to grafana/datadog etc)?If yes, what metrics would we like to capture?

Toggle QA Checklist

- [ ] Create test plan - **Template: QA Test Plan** - [ ] Identify test data needs - [ ] Gotten & incorporated test plan review - [ ] QA Complete - [ ] Waygate tasks complete - [ ] Added new feature cases to the RC in TestRail - [ ] Added feature cases to appropriate section of active cases in TestRail - [ ] Spun up a (future) UI automation ticket for new feature

[bischoffa]

Talked about during 9/19 SOS

[TimRoe]

There's more than a few examples in the recent March/April app store review summaries, but this one stuck out to me as emblematic of veteran willingness to provide feedback when we allow them:

This app is helpful but it should include the link to mileage travel for the medical appointments. I would have told the developer that, but it sends you to general va help if you ask for app assistance. The chatbot will not know what I am asking for VA. Help us help you (to help us).