Groom 10203 KPIs for KPI dashboard

joanneesteban commented 4 years ago

Issue Description

Groom team KPIs for technical feasibility.

Tasks

[x] Groom analytics for technical feasibility
[x] Document data sources for each metric
[x] Document unavailable vs. available metrics
[ ] If any, add implementation issue for additional necessary implementation

Acceptance Criteria

[ ] VSP Analytics & Insights understand which KPIs are feasible, how to retrieve them, and has a documentation of blockers for additional KPIs

KPIs in Epic.

jonwehausen commented 4 years ago

Update 9/4

Amy Knox requested items related to the conversions of the form and ineligibility alerting
As a result we drafted an open and closed funnel solution for form 10203
Stoop up a mockup dashboard at https://va-gov.domo.com/page/1767988090 to include each of the funnels with 2 different visualizations for each.

For an explanation of the closed funnel vs open funnel:

The open funnel chart is flexible in that: the user can enter at any stage, and accounts for pages that can be skipped or not shown at all (i.e /confirm-eligibility). For example, in cases where a user doesn't reach /confirm-eligibility, that step is still counted towards funnel execution. The conversion rate is agnostic of whether someone reached the ineligibility alert or not.

The closed funnel charts ensure the user was forced to reach every step of the form. In other words, the closed funnel charts ensures folks definitely reached /confirm-eligibility and carried on with completion. This funnel type may undercount conversions if users don't follow those steps precisely.

Current limitation to both funnels to note: These charts enforce the user started and completed the form within the same visit. In other words, if someone started the form, saved, came back 3 days later and completed those would not be counted. We are working on follow-up queries to account for these scenarios.

amyknox commented 4 years ago

@jonwehausen - thank you for putting this together!

A couple questions...

How can we tell the date range for this report?
Can the date be customized?
Could the closed funnel be customized to exclude the 1st eligibility alert from the funnel?

If people see the 2nd alert, by definition they would have seen the 1st one, too. So if we could exclude that, it'd really show us everyone whose responses indicated they were ineligible but applied anyway. The view with both alerts is showing us just the people who are ineligible due to the benefit they're using.

amyknox commented 4 years ago

@jonwehausen

Another KPI identified in the 10203 product outline is "Successful 22-10203 STEM Scholarship Application submissions esp. compared to applications via 22-1995" This is reliant on reporting on COEs issued by the team in Buffalo - would there be a way to incorporate their external data into the DOMO dashboard?
Interestingly, there was a rule change last month that relaxes how eligibility is determined so the approval rate for STEM via 10203 will get an unexpected boost

joanneesteban commented 4 years ago

@amyknox Hi! Can you elaborate on the COEs from Buffalo? Where does the data live, and can you let us know how the KPI is reliant on this?

amyknox commented 4 years ago

COEs are Certificates of Eligibility which are issued to applicants who are approved for the STEM scholarship. The data would live on the system that the folks in Buffalo work with directly - not something that is GA trackable.

The KPI relies on this as these are individuals who complete a successful submission AND are found to be eligible for the scholarship. As you may recall, a key issue with the STEM scholarship application has been that the team in Buffalo has been dealing with an over-abundance of ineligible applicants.

joanneesteban commented 4 years ago

@amyknox Thanks for explaining!

For us to incorporate this data, can you share where the COE data lives and if it has an API we can use? *

Also, would you say that your KPI is: Successful 22-10203 STEM Scholarship Application submissions = Eligible applicants / Successful VA.gov 10203 submissions Is this different from the KPI, Valid 22-10203 Application submissions, # and %

amyknox commented 4 years ago

I'm looking into COE's. Each individual who gets approved for education benefits gets a COE. I'm not optimistic there's an API to track that but will let you know what I find out!

For valid vs. successful submissions, we should find an agreed on vernacular. Here's my thinking... "Successful submissions" are those we can track solely in Google Analytics. An applicant enters the funnel, completes the application and submits the form. TA-DAH - success!

"Valid submissions" are those where the application is submitted AND the applicant is actually eligible for the scholarship. This will be a lower number and require input from the Team in Buffalo - or pulling from that COE API, if it exists.

Let me know if this makes sense to you!

bmcgrady-ep commented 4 years ago

10203 KPI	Metric	Source	Go/No Go
Successful submissions	10203 successful submission total events vs 1995 successful submission total events	BigQuery	✔️ Go
Valid submissions	Total 10203 successful events + COE	BigQuery	🔶 Future Go

Note: We will need the COE data either from the team in Buffalo or through an API for the Valid Submissions metric.

joanneesteban commented 4 years ago

Thanks for your work on this, @bmcgrady-ep ! @amycesal please review the current dashboard.

@amyknox let us know if you have any luck with the API!

jonwehausen commented 4 years ago

@bmcgrady-ep Could you advise what the next steps are here towards closure is here? Thanks

bmcgrady-ep commented 4 years ago

@amyknox - I am closing out this issue. Feel free to reopen this issue, if you get access to the API, thanks!

department-of-veterans-affairs / va.gov-team