Analytics Reporting Request for Caregiver Team (Health care application)

[jonathanjnelson]

Metrics Reporting Request

What this form is for

Use this template to request one-off performance, usability, or outcomes va.gov metrics that is difficult to access or that needs analytics interpretation.

Insights will usually provide data reports within the current sprint, but if there is time-sensitivity, please note them in "additional comments."

Instructions

Please fill out the following sections:

• [x] Description
• [x] Timeframe for Analytics Requested
• [x] Analytics to Track Down
• [x] Which Product Questions are you trying to answer?
• [x] Additional Comments

Description

Who is this request for?

• [Jonathan Nelson]

• [Alayna Abel]

• [Ming Ligh]

  Send Slack message to DSVA Slack Channel, #VFS-Platform-Support, that includes: email address(es) for user(s) requesting data and link to this Github issue.

• What team/project is this work for? this adds context for the data

  • [Caregiver Team and Research plan for Health Care Application 10-10EZ)

Timeframe for Analytics Requested

Please provide the timeframe for the metrics you are requesting. What timeframes are you looking to analyze (e.g. before vs. after)? I will look to you to give me a suggested timeframe. That will provide accurate picture with the data.

Analytics to Track Down

Average completion time How many people start the application and don't complete it? How many people visit "intro page" but do not start the application? Where are users coming from? What devices are users using?

Which Product Questions are you trying to answer?

The health care application (10-10EZ) was one of the first forms onboarded to the new VA.gov platform.

We want to discover what pain points Veterans have with the current health care application, and understand why many Veterans prefer to apply in person rather than online.

Other Helpful Questions to think through

• Who is your audience?
• Why do you need this data?
• What are you hoping to find?
• What is your hypothesis that you're hoping the data will support?
• What area of the VA are you hoping to assess?
• How frequently do you need this data?

Additional Comments

Add any comments or questions for the Insights team here (if applicable).

This would be great to get as soon as possible

Please leave the following blank

Acceptance Criteria

• [ ] The data has been identified and documented in this ticket - items that we don't have access to are noted with cannot be accessed
• [ ] The metrics are shared back with users.

Definition of Done

• [ ] All appropriate issue tagging is completed
• [ ] All AC completed

[jonwehausen]

Hi @jonathanjnelson ,

acknowledging receipt, thank you in advance and we will be in touch with any questions

[joanneesteban]

@jonathanjnelson @ala-yna thanks for creating this. Do you have a product outline for 10-10EZ started?

[jonathanjnelson]

Hi @joanneesteban we have this Project Outline for our initial 10-10EZ work. The team wanted to broaden this work and start by looking at some data. Let me see if I can track down another Project Outline.

[jonathanjnelson]

@joanneesteban Sorry for my delay! You can see a rough draft of the Project Outline. You can also see the Project Kickoff document here. Alayna also made sure I have access to Domo so I am going to dig into that soon!

[jonathanjnelson]

@joanneesteban Is it possible to schedule a time to talk through Google Analytics and Domo (both of which I have access to) in order to gather some data on the Health Care Application (10-10 EZ)?

I am on PTO October 26-30. Perhaps we can schedule something for the week of November 2?

[mlighdc]

Hi @joanneesteban and @jonwehausen, as you know Jonathan is on vacation this week. I am checking in to see how this is request is progressing. Thanks!

[joanneesteban]

Notes from 11/19 Meeting

The Caregivers team is looking to share findings on Monday, 11/23.

Metrics to Look At: @bsmartin-ep @JasonTheBearded Time frame - 2 views:

• March - October of 2019 and March - October of 2020
• July 1 - October of 2019 and July 1 - October of 2020

How many Veterans are looking at the How to Apply page?

• Number of veterans looking at the How to Apply page

• GA

• Questions/research for Caregivers

  • It looks like seasonality between 2020 and 2019 is the same. For the dip that occurred in 2019, that's due to deduping submissions in GA. 2020 online submissions & people going to the How to Apply page went down ~10% from last year. Why might this have been the case? Did VA.gov traffic go down as well, and 1010 numbers just reflect that, or did fewer veterans find the form? Were more people applying offline? How did the Mission Act change how veterans applied for and received health care benefits, and might this have affected the numbers?
  • If offline applications went down, online applications should have gone up (assuming that veterans still needed health care, veterans are not getting the care or benefits they deserve). It's probably safe to assume that if offline applications went down, but the same or fewer veterans are looking at the How to Apply page this year, then the veterans who were applying offline are not looking at how to apply online.

How are Veterans getting to the How to Apply page?

• Number of veterans broken down by entry to the How to Apply page. Looking for data prior to April 2020

• what does direct/none mean in the Domo dashboard? - Healthcare campaign dashboard

• Questions/research for Caregivers

  • Are there more veterans coming from another page on VA.gov, Google, or another source? This can help with where to allocate resources to market the form
  • During Covid data monitoring, the Analytics team found time and again that spikes in VA.gov traffic and specific health page traffic increased drastically due to VA Notify/VETexts, and that VA blog or social media posts were causing traffic to spike. These marketing efforts can help increase findability of the How to Apply page. The next question may be, if you increase traffic to the How to Apply page, then are you increasing the number of submissions? Or do you need to look into conversion from the How to Apply page to the form itself as well?

How many Veterans are completing and submitting the form online?

• Number of successful online submissions - The Domo funnel has different numbers than the GA funnel

How does form completion compare to other forms on VA.gov? (for Caregivers, which forms do you want to compare to?)

• Funnel conversion compared to other forms
• Product Health MVP, but larger time frames

How many Veterans start the form but don’t complete it?

• Healthcare campaign dashboard

Are there any points within the form where Veterans tend to get stuck?

• Healthcare campaign dashboard

• Is there a save in progress or a way to track where veterans have left the form and come back to it?

• Questions/research for Caregivers

  • There's only 20% of users going from the How to Apply Page to the form itself. Is this because they're going to get the info they'll need for the form and come back to it later? Are you looking to increase the conversion from How to Apply to the form itself? Or the findability of

[mlighdc]

@joanneesteban based on the notes ^^. Can we see how many veterans come back at a later time to complete the form? I thought there was a measure in GA that has this information but I can't remember exactly.

[ala-yna]

@joanneesteban, do you have an estimate on when your team will be able to get some of these questions back to us?

[joanneesteban]

@mlighdc we will need to do additional implementation for that metric; we're unfortunately a bit behind on resources for the next few weeks, and will need at least a couple of weeks (a sprint) to get you the entirety of this information.

[joanneesteban]

1010EZ Funnel in Google Analytics

[mlighdc]

@joanneesteban if I have specific questions about the funnel and other GA data, who can I reach out to for a quick conversation?

[joanneesteban]

@mlighdc thanks for reaching out - you can set up a meeting with myself and @bsmartin-ep .

[bsmartin-ep]

@mlighdc -

Dashboard updated with User-based funnel and data table:

https://va-gov.domo.com/page/290380612

[mlighdc]

Thank you @bsmartin-ep! Looks great!!

[joanneesteban]

Removing @JasonTheBearded from this ticket and adding to the formal KPI dashboard request for the Healthcare app.

[joanneesteban]

^some of the charts may be duped

[joanneesteban]

@ala-yna @mlighdc can we move this issue to closed? Jason has begun work on the KPI dashboard noted in #12460.

[joanneesteban]

Hi @ala-yna @mlighdc - we're closing this issue & the KPI dashboard work can be tracked in https://github.com/department-of-veterans-affairs/va.gov-team/issues/16547