Recommendation to VA.gov team: KPIs for Mobile App Promotional Banners

[g-opzt]

Acceptance Criteria

• Documented list of KPIs in alignment with the Brief (linked below) that are possible including:
  • how they will be measured
  • any calculations performed

Background

• Mobile App Promotion Banner Initiative Brief
• Ticket: Promotional banners for mobile app on mobile-breakpoint Hub pages

See above doc & ticket for VA.gov team's context

Method

Use data analytics cycle define and plan steps to define KPIs that can be measured via data available to the web and mobile teams combined. #6095 covers discovery work about what is possible in terms of new data collection.

[g-opzt]

Initial pass at the questions to be answered below. Will be refined through the method in ticket description:

• How many iOS users who already have the app on their phone are opening it from a VA.gov promo banner?
• How many iOS users who don't already have the app on their phone are downloading it from a VA.gov promo banner?
• How many login fails are they experiencing?
• Are they completing mobile app tasks related to the content of the VA.gov url they were direct to the app from?

[rtwell]

@g-opzt a few notes for this work—I would recommend we concentrate this ticket into what the mobile app team needs. PW can manage their own tickets and analytics.

How many iOS users who already have the app on their phone are opening it from a VA.gov promo banner? How many iOS users who don't already have the app on their phone are downloading it from a VA.gov promo banner?

for both of these, according to the initiative brief, these are not MVP, unless this data is already available. If it isn't, let's create an issue to figure out how to get the data and make it available on our dashboard and the mobile team will prioritize accordingly. I think this might be #6095—please confirm.

How many login fails are they experiencing?

This is not related to this funnel. Please create an issue that captures the Mobile App Team's need to capture this data.

Are they completing mobile app tasks related to the content of the VA.gov url they were direct to the app from?

It's my understanding that we require deep linking to get this definitively, which we have an issue tracking this for Q3. What pub websites has documented in the brief is our closest proxy to that. the public websites team is not concerned with task completion—that is the mobile app team's data to be concerned about and we should capture and document it for ourselves. (See letter downloads in the brief for an example). I would update the AC of this issue to

• In the product brief, reference the table MVP KPI Measurements, in the column labeled "Link to data source" and validate that the data points identified map to the product KPI in the first column.

Since we don't have deep linking now, let's not make perfect the enemy of good.

[g-opzt]

@rtwell sharing here a more detailedd version of what I discussed with Rachel in DMs when she asked about the relevance of this work before we moved to the thread in #va-mobile-app-analytics

If we don't do 6095 and 6089, we will only be able to answer the following question:

• Is the number of iOS users going up (in total and per specific app events)?

We already know it's going up because our user base is growing every month, and that rate has varied; this number alone will not tell us about the promo banners influence on that number. This isn't an MVP measure because it not measuring whether or not the banners are leading to downloads/opens--it mixes together every possible way users find the app.

If we complete #6095, we will know if we are able to answer at least some of the questions discussed above (or similar) without deep-linking. I will clarify in that ticket that it is not considering use of deep linking at all.

6095 is the ticket to confirm that and get the technical details correct for implementing it. It could be that we discover blockers to the non-deep linking approach but I can't know that until I do the work. If we at least complete #6095, we'll have uncovered any possible (without deep linking) accurate measures of the traffic from banners to the app. Whatever options we do find can be triaged down to the smallest possible lift.

If it turns out we don't have anything after #6095 or if we choose to not do this work, then there isn't a real indicator of performance for these banners increasing our overall user base or the number of users completing a task in the app.

I know @Hallm13 is following up with @rachelhanster next week so I look forward to hearing the direction of this work once you've all had a chance to discuss and align.

[bischoffa]

Moving to backlog per PO direction and discussions on focusing on just creating the urls and tracking download through app store.