In-App Veteran Recruitment - Implementation

[Hallm13]

Project Status Overview

Project Objective:

The recruiting company (Perigean) VA uses to recruit Veterans for user research studies has given us a unique URL to track mobile app users so we can prompt folks with in our app to sign up for feedback studies. Using the unique URL, they can tag those users as coming from the VA Health and Benefits app.

The current method for us to identify users of the app relies on Veterans self-reporting that they are users—this is unreliable because Veterans often misreport, which can make conducting research with app users challenging.

Current Dependencies / Blockers

Blockers

|Blocker | Team / Owner | Est to Resolution | |-------|---------|-------| | Implementation to be prioritized | PO | No ETA | | Approval to move this into production from Danielle / Kevin | Mobile | No ETA as project itself is not prioritized | | | |

Project Timeline

Milestones

|Step | Timing | |-------| ---------| |Sprint 0 - Discovery | Sept 2022 | |Step 1 - Strategy | May 2023 | |Step 2 - Design | June 2023| |Step 3 - Building / Implementing | Pending PO prioritization | | Kevin / Danielle Pre-Prod Approval | | | 100% Release | | |Step 4 - Post Verification| |

Monthly Update

Updates

|Month| Project Health | Progress & Key Accomplishments| Issues/Risks/Blockers| |-------| ---------|--------|---------| |June '23 | Green | MVP design and content defined, Step 2 complete, just needs prioritized | July '23| Q3 Alternative project, awaiting PO prioritization | Not prioritized | |Aug '23| On Hold | Q3 Alternative project, awaiting PO prioritization | Not prioritized | |Sep '23| On Hold | Q3 Alternative project, awaiting PO prioritization | Not prioritized | |Oct '23| | |

Detailed Project Scope

Problem Statement

• Currently, it is difficult to reliably recruit participants via Perigean that are actually users of the mobile app. We are looking for ways to prompt people who are already using the app to self-select in a way that Perigean can track.
• High miss rate as Veterans say they are using the VA Health and Benefits Mobile App but are just on the mobile version of the va.gov or using a different app
• Have seen those recruited are relatively new to the app and don't know a lot about it

Assumptions and Risks

List of Assumptions and Risks

- We will not be pulling Veteran information into the form (address from app to address field) - We will not be hosting a form in the app - We will only be serving a link to the Veteran (link is from Perigean) - Inability to match unique a Veteran clicking out of the app and taking action on Perigean's web - Inability to modify flow, experience outside of the app within Perigean's ux - Inability to prevent Veterans who have already signed up with Perigean from not signing up again - Inability to unique match if a app user completed the Perigean form - Inability to modify Perigean UX - Mobile is not hosting a form in the app - Link is not a VA endorsement - Will need Perigean to report on metrics for this feature - Perigean may have copy / approval processes that need to be followed - Taking Veterans to a 3rd party site may lead to spam / phishing concerns - Veterans signing up to participate through the mobile app will improve participant recruiting criteria accuracy. - Veterans want to participate in user research studies and will be open to registering for studies while using the app. - We can recruit for Perigean through the VA flagship app. - The Perigean form isn’t too long/difficult for participants to complete on their mobile device.

Definition of Done

Following needs to be true

|Item | Completed | |-------| ---------| | | | | | |

Project Scope

Items in and out of scope

|In Scope | Out of Scope| |-------| ---------| | Accessibility friend, human centric, mobile design | pre-populate perigean form w VA data | | Business logic to control when it is displayed and not displayed | create enrollment form inside the mobile app | | Serving the Veteran a link which takes them out of app to enroll | Appending Veteran features to the enrollment |

Important Project Links

Links

- [Sprint 0 completed Sept 2022](https://docs.google.com/document/d/1Ykx0CmDNF1kMEjPeKRokiy0bRG81iquR5QGy5hMGGGk/edit) - [Step 1 Product Canvas](https://docs.google.com/presentation/d/1dN8T93Uc_SDh8cZEJUbjYvEjFFI0K7C9aJztL_tn0s4/edit#slide=id.g201aab40dc3_0_0) - [Perigean url link / form for in-app recruitment](https://docs.google.com/forms/d/e/1FAIpQLSfRb0OtW34qKm8tGoQwwwDFs8IqwOMCLTde3DeM-ukKOEZBnA/viewform) - [Figma Design - working file](https://www.figma.com/file/L702ULyDKZbO2WBIHdJbyX/%F0%9F%94%8D-GlobalFeatures2.0---Working---VAMobile?type=design&node-id=807-3292&t=pg7Wq68qQRx3aYLg-0) - [Figma Discovery](https://www.figma.com/file/WpxFED4wlfbKD4LEqP9Pwm/Discovery%3A-In-app-recruiting-and-feedback-collection?type=whiteboard&node-id=0-1&t=LmhvJVjNc45zM1IA-0) - [In-App content info from Misty/Danielle convo on 6/21](https://www.figma.com/file/pTJ90KbVDoutPIbGjQ1y7o/Misty's-Sandbox?type=design&node-id=909-8020&mode=design&t=QKa5x2MGypdx1Mid-0)

Team Members

Contact List

- **OCTO Product Leads:** Ryan Thurlwell - **Ad Hoc PM:** Adam Kasmier - **Engineering:** Jayson Perkins and Jon Bindbeutel - **Design:** Lauren Russell - **Content:** Misty - **External Team 1 & Role:** Danielle Thierry - **External Team 1 & Role:** Kevin Hoffman

OKR and Analytic Metrics

Metrics

|What we are measuring | Why | Event Name |-------| ---------| ----- | | | | | | | | |

Questions

Open questions

Closed

| Questions | Response| |-------| ---------| |- Identified mid June, that using Perigean link to enroll Veterans may be considered an endorsement by the VA. Lauren Alexanderson will need to weigh in, Liz sent this concern/question to PO Ryan who is supposed to get back with Mobile the week of 6/12| resolved | |-Asking Kevin/Danielle for approval for release timing |yes needed pre launch | |Is there a compliance concern with telling the Veteran they will get $5.00 when they won't if they already signed up - investigating | no; however mobile is not including specifics or details around money Veterans can earn | | Given we are going the direction of using an alert box to market to Veterans to sign up - we need to consider logic and rules for overlapping with App Update Alert and Whats New In App Alert - should use same logic as to only show 1 of them at a time vs showing many | app update alert is for a small subset; whats new in app we control post release - so we will plan with this in mind if / when an alert box communication for research goes out | | What reporting can Perigean provide ? | Liz connected with Brian Moon; Perigean is limited in their own reporting and can only see successes (not attempts, fallouts, etc given Google form limitations) slack convo in comments |

Checklist

Toggle Product Checklist

**Project Kickoff & Requirements Gathering** - [ ] Project Kickoff session - [ ] Initiative Scope finalized - [ ] Link Product Brief - [Sprint 0 ](https://docs.google.com/document/d/1WHDJ1iCt9J-acLx8_DSL4tkRKoVYqJGaRUqXuFf7ceg/edit# ) - [ ] Link to Implementation Strategy - Implementation Plan - UAT Plan - Release Plan - Metrics Identified (what metrics are we capturing, how do they tie into stakeholder OKRs, etc) **Pre-Production Readiness** - [ ] Associate Readiness - [ ] Update product guide - [ ] Coordinate with contact center support teams - [ ] What’s New Content - [ ] App Store/In App messaging (if applicable) - [ ] Coordinate with Release coordinator - [ ] App store Content changes required? Y/N - [ ] App store images needed? Y/N - [ ] QA Signed off - [ ] UAT complete - [ ] Obtain Go/No Go decision **Feature Go Live** - [ ] Ensure Waygate/Feature toggle release ticket submitted w/engineering & QA - **Template: Turn on Waygate/Turn off feature flag** - [ ] Coordinate with content & release coordinator if what’s new in app is required

Toggle Team Collaboration Checklist

- [ ] Design Intent & Feasibility discussions - [X] Research needed? No - [ ] UAT Needed? Yes/No - [ ] Ticket creation - [ ] Demo feature (mid check, end of development)

Toggle Design/UX Checklist

- [ ] Review problem/opportunity statement - [ ] Add questions and assumptions to product brief - [ ] Add research links to product brief - [ ] Document high-level research/design plans - [ ] Create lo/hi fi wireframes & user flows if applicable - [ ] Add links to Epic & applicable GitHub folders - [ ] Socialize designs - [ ] Sign off received - [ ] Complete Design QA - [ ] Does it require component review? - [ ] Accessibility review needed?

Toggle Content Checklist

- [ ] Review product brief and surface any questions, assumptions & risks - [ ] Participate in project discovery and kickoff activities/ceremonies - [ ] Complete comparative analysis/content research activities for net-new content - [ ] Review current content and determine where app may need to differ and potential improvements to share with web - [ ] Review past VA research and decisions documentation - [ ] Support Research & UX Design in research sessions and synthesis - [ ] Make content recommendations for lo-/hi-fi wireframes in collaboration with UX Design - [ ] Collaborate with Sitewide Content team for alignment, improvements, and sign-off - [ ] Participate in FE hand off - [ ] Complete content QA - [ ] Write copy for What's New In App - [ ] Write copy for app stores' What's New sections. - [ ] Write copy for updated screenshots in app stores

Toggle Engineering Checklist

- [ ] Are UI designs/ specs available/ready? If yes, they should be attached to attach specs. (We are assuming the design specs have been approved) - [ ] Are there any deadlines that need to be met? - [ ] What is the roll out plan? (Do we need a feature toggle? is this going to be available to every user immediately or do we want to roll out to a few users first) - [ ] Does this project require api integration? if yes, what api endpoint are we integrating with and what is the expected response body? - [ ] Do we want to capture analytics for this projects (i.e. update GA, add logs to grafana/datadog etc)?If yes, what metrics would we like to capture?

Toggle QA Checklist

- [X] Create test plan 6376 - [ ] Gotten & incorporated test plan review - [ ] QA Complete - [ ] Waygate tasks complete - [ ] Added new feature cases to the RC in TestRail - [ ] Added feature cases to appropriate section of active cases in TestRail - [ ] Spun up a (future) UI automation ticket for new feature

[bischoffa]

Sprint 0 completed 1/24 https://docs.google.com/document/d/1ieONNnsObXzTBt6GlIFBVvuJL7-fnlFzQWWMRrZV5gI/edit#

[bischoffa]

Rough outline of step from Jen's perspective shared on 6/2

• Check with Perigean to see if we have to use copy they give us or not. If not, Liz can take a first pass. - 2 points
• Look for examples of how other apps handle this type of pattern (probably on mobbin.com) - 2 points
• Find out how we currently handle this type of pattern in order to make a recommendation. So review our sending-folks-out-of-the-app prompts as well as our App Store review prompt. For App Store prompt we might need to check with FE to see - when does it show up, frequency (if any), what does the patterns look like, and where do we show it (for the when, Binny did a quick check and it looks like 122 days but said there’s several metrics for it) - 2 points
• Determine what metrics we can/want to track on our end ?? (This would be coordinating with Greta so not sure their capacity or this lift)
• Get Misty to approve/edit content - 1 point (note that if Misty edits Perigean content we’ll probably need to run it by them)
• Mock up up screen and flow and hand off for FE - 3 points
• VQA once QA passes - 1 point

[bischoffa]

Matt approved on 6/6 is moving forward and wrapping up all of Step 2 for this work. This would allow the project to be completed defined and literally ready to be worked by engineering when it was prioritized which POs indicated to Jen they would like it in Q2 and for Q3.

UX work needed to completed Step 2 would be done if UX had sprint capacity in Sprint 73. Post planning we had enough to do so however was asked by Jen to hold off as the Department Managers needed to talk to the POs to see if that is how the want to spend the extra UX capacity for this sprint. Waiting update from 6/7 meeting on how to proceed.

[bischoffa]

POs approved during Demo on 6/8 to complete step 2 for UX work given extra capacity. Both Ryan and Rachel expressed excitement and interest in the project and feel its a solid win without a ton of effort and sets up well in the future.

[bischoffa]

UX work being pulled into current sprint given this approval.

[bischoffa]

1 page review of the work can be found here:

[bischoffa]

SoS update

• In-App Recruitment Step 2 (Adam)
  • Thanks Ryan for confirming we can use the Perigean link; Danielle will be included in content approval- Slack thread
  • Dependency - We will need Kevin Hoffman / Danielle Thierry approval before implementing as they are not currently looking to increase their recruitment database size (see Slack thread for more details)
  • Proceeding with Step 2 focusing on UX design and business logic per Po's approval last demo

[bischoffa]

6/15 discussion during In-App Touchbase - Jen, Holly, Ryan, Adam

Decision points:

• Ryan said we should proceed with what design thought was best
• Ryan agreed this should live in a permanent place in the app
• Ryan would like the content modified and simpler on the registration page
• Ryan would like us to explore adding language around Perigean's $5.00 enrollment

Few open items on this:

• Perigean feedback on $5.00 language being used, especially for those that already signed up and got the $5.00 - feels like a compliance concern to promise $5 to people who will not get it as they already got it - I asked Jen on 6/16 to followup with Perigean
• Review with engineering on webview vs taking the Veteran out of the app (Adam met with Binny on 6/15 - believes that the program has typically tried to keep the veteran in app, also the monitoring/measurements are the same as we do not see intent either view)
• Review with analytics on webview vs taking the Veteran out of the app (Adam met with Greta on 6/15 - only difference in metrics is that for webview we should be able to differentiate between the link leading to an error or getting to the page successfully) also discussed other options for future consideration like different urls for different targeting / placement, if we had the ability to pass a unique id from app to Perigean to track end to end registration,

[bischoffa]

Misty reviewing proposed copy for final review with Danielle on 6/22. Misty already worked with Danielle earlier in the week to get copy which led to a mockup of suggestions and us proceeding with copy 3 image below.

Link to Misty's sandbox with the info/discovery with Danielle

[bischoffa]

Identified as a Q3 2023 project, however its an alternative project meaning its not a Q3 focus but is rather a project that could be pulled in if there is bandwidth or delays in other Q3 priorities. Work not getting done in Q3 is not deemed critical or a failure but more of a possible stretch goal.

[bischoffa]

[bischoffa]

Slack thread on PO convo regarding copy feedback

[bischoffa]

Alternative project thus no estimated timing can be added to the card

[bischoffa]

Alternative project for Q3 however it was never prioritized by the POs for Q3 so the project will just sit and wait until its prioritized to be implemented.

[bischoffa]

POs never prioritized this alternative project for Q3 thus it was never implement. Project has Step 0 - 2 completed. Only Step 3 (implementation) remains.

Not sure where this falls into Q4 planning. However this project only needs external approval to release, all the build work is contained within mobile.

[bischoffa]

Recommend using the start date for this when it gets picked up. Technically this started back in Q2 I believe but assuming we don't want to use that date.

[TKDickson]

@ajsarkar28 does it make sense to split the "need changes post HSP" tickets into a separate epic and close this one?

Either way, are those tickets part of home screen MVP scope?

[rtwell]

Perigean fixed the issue of recruiting participants with the app so the benefit of the Give Feedback button is likely moot.

[rachelhanster]

To reduce confusion within OCTO, here's a summary: This ticket was done as of early March 2024. It was difficult to accurately recruit research participants who have and use the app. So, in the Profile, Settings section of the app, a "Give Feedback" link was added. The link launches an embedded Google form to sign up for research with Perigean. Later in 2024, Perigean appears to have fixed the issue of recruiting actual users of the app. This needs to be validated. If true, benefit of data from "Give Feedback" form needs to be revisited.